As often on this kind of post, I would like to remind you that the content you will find here is for educational purposes only.

Unauthorized intrusion in an information system is punishable by fine and/or imprisonment.

Understanding attacks means knowing how to avoid them. In this post, I propose you see a common network attack model: ARP poisoning.

What is ARP?

To understand the attack, we must already understand what it is based on.

For ARP, we will start from the OSI model, describing the different layers of the network.

ARP is the link between layers 2 and 3, it is what will allow a computer or server to convert an IP address into a hardware address (MAC).

To work, ARP uses a broadcast system. Each device on the network sends its IP address associated with its hardware address to the whole network.

You can easily access and modify this table with the arp -a command (either under Linux or Windows).

The weakness of this protocol

I think you have already located the weak point: each node sends information, and there is no way to check the veracity of this information.

So, I can very well advertise my MAC address as the router's IP.

Each node (or nodes targeted) on my network will receive this information and update its ARP table with the match.

How does the attack work?

There is a prerequisite, to be on the same network as the target.

Then the idea is to be between the target and the router. This will allow to launch a "Man in the middle" attack to capture all the network traffic between my target and the router.

For example I will describe in this blogpost, I will put myself between my PC and my Livebox on my own network.

To do this, I will send two different pieces of information.

From my attacking PC, I will tell my livebox that I am the victim's PC.

To my victim, I will say that I am the livebox. In this way, I can have both information flows passing through my attacking PC, positioning myself in the same way as a proxy.

The attack in detail

Requirements

As I described above, I will need several things to proceed with my attack:

• Access to the same network as my target
• A PC with different tools (which I will describe below)
• The IP of my target, even if I can do without it, as we will see

In the case of this post, I will use two "Lab" virtual machines:

• My attacker will be a VM running Kali Linux
• My target will be a VM running Windows 10

Detect the Windows PC

To perform my attack, I will use the popular bettercap.

It will provide me with all the tools I need to detect the devices on my network and perform ARP poisoning.

So I will start it with the command below:

bettercap -iface eth0

Bettercap works with modules, which can be configured and executed. I won't go into the details of the modules here, and I invite you to consult the associated documentation for more details.

If necessary you can use the help command to see the available commands and modules and their status.

At first, we will use the "net.probe" module.

As its name indicates, this module will allow us to detect the devices on the network.

net.probe on
net.show

We find my target below, which is none other than the IP 192.168.109.130

Let's start having fun

Now we're going to launch the attack on the target's ARP table.

Before we start, I'll just capture my target's table.

The most important thing here is the IP 192.168.109.2 which is the IP of my virtual router.

From my Kali VM, still in bettercap, I'm going on the offensive!

set arp.spoof.fullduplex true          #Launch the attack on both sides : victim and router
set arp.spoof.targets 192.168.109.130  #IP address of the victim
arp.spoof on                           #Initiate the attack

The first line is important, because it will allow launching directly the attack on both sides.

Note that some routers are protected against ARP attacks. In this case, we will only see the frames leaving the target, but not the return, which limits the attacks.

If I look at the ARP table of my target now, we'll see that the router's IP is not the same anymore!

By the way, we'll notice that it now has the same MAC address as my Kali VM 192.168.109.129.

Now I am able to see the traffic going in and out of this PC with a simple net.sniff on

What does it get me at this point?

At this point, I am able to capture all unencrypted traffic leaving my victim's PC.

This means I can see:

• HTTP calls
• Unencrypted DNS requests
• SNI requests

It's all crap, we only see HTTP

Many will say that this is a small part of the network. This is true and false at the same time.

If I'm in a company, many consider that it is not necessary to put encryption for internal endpoints for example, so it can potentially capture a lot of traffic.

Also, knowing which site a user is going to potentially allows me to target an attack, whether it's phishing or social engineering.

Finally, this part should be seen as a first step, indeed, once I am positioned in man in the middle it also allows me to alter the traffic to change what my victim sees, but we will see this point in another post!

In conclusion, beware of ALL networks

The purpose of this post is to remind that an attacker is not necessarily external, even in your corporate network it is possible to do a lot of damage.

ARP spoofing is one of the issues that are normally covered by most enterprise routers, but not all. It is important to protect against it.

Most consumer routers don't offer any particular protection against this, for example, I can do this attack on my Livebox.

This also reminds us of the importance of encrypting all outgoing traffic from your PC, via a VPN for example, there are still many (too many) services communicating in HTTP.

In companies, do not hesitate to impose TLS everywhere, including internally.

Unless you live in a cave, you couldn’t miss the last few days the flaws discovered on the Java log4j library.

I’m not going to make another post talk about this flaw, but rather to talk about my experience in the field on the impact of this flaw at the operational level.

The importance of having an up-to-date inventory

I talked about this on Twitter (in French), but for me, this flaw highlights the fact that many companies lack an up-to-date inventory of their resources.

When a breach like this occurs, it is important to be able to quickly identify which resources are at risk. But having an up-to-date inventory is not just about the machines, but also about their content.

This is even more important when you exploit the power of the cloud, having volatile machines make their management even more complex.

For that there are several solutions.

Doing only infra as code

This point will seem essential to many of you, but only doing infra as code simplifies things enormously in order to quickly visualize what is being deployed, on which project, which machine, which account, etc.

In a CI/CD pipeline, it is the role of a SAST (static application security testing) to identify common flaws when compiling new projects; however, there needs to be tooling for already compiled projects, for example, on GitHub, Dependabot showed me fairly quickly the projects in which it had identified the use of the library.

However, this point is clearly not enough. Just because I don’t explicitly define a library doesn’t mean it is not used.

Scanning already created artifacts

If your applications are already compiled, there are often solutions on your library managers to scan them.

For example, Docker Hub allows you to scan your images for the library.

However, this is still not enough. On your servers, you do not necessarily use only packages that you master and reference yourself.

For example, you can use an AWS image of a publisher in the marketplace, or have third-party applications deployed in your infrastructure.

Continuously scan your servers

This is where the third level of scanning comes into play: you can continuously scan your machines to quickly identify servers in danger.

To do this, there are tools like Nessus Tenable or Rapid7 InsightVM. The role of these tools will be to scan your servers and report to you the machines at risk.

A simple search with the identifier of a CVE allows you to quickly identify the impacted servers.

How I lived this stage

Like many in infosec these last days, this first step was complicated. Even with tools, there is always a risk of missing some resources.

I work in a small team and we reacted as best we could, but given the risk involved with this vulnerability, we are re-running several checks. This point is exhausting, because we have to do a lot of manual actions, not everything can be automated.

It is a laborious step, mentally exhausting, but essential and critical.

Once the servers are identified, what do I do?

Now that I know which of my resources are affected by the log4j vulnerability, what do I do?

The first thing we often forget: don’t panic!

This is a behavior I often see during security incidents, yet it is all the more important to keep a cool head as a mistake can be costly!

Patch, patch, patch…

The solution will often be the same here: patch or update the application. However, not all editors are as serious about this game.

At this point, I have seen several behaviors:

• Publishers who are very reactive and transparent about the risks
• Publishers who say they don’t know (which is not necessarily reassuring, by the way).
• Publishers who say they are impacted, but take several days to release a version with the necessary fix
• Those who say they are not impacted and then indicate the opposite
• Those who prefer to indicate that they are not impacted when they are far from java (for example HashiCorp and its tools written in GoLang)
  

Depend on other teams

At my position, I am not the one who patches, but in the team that will identify the risks, measure them and ask the necessary actions to other teams (Dev, Ops, DataOps, etc.)

It’s a position that can be frustrating, because you depend on people over whom you have no power of prioritization and not everyone necessarily perceives the risks of these flaws.

This is why it is important to have (once again) a pedagogical posture, and to explain the risks and why it is important to react quickly.

And it is also at this stage that it is important to trace all the requests and do the associated follow-up.

One of the temporary solutions can sometimes be to mitigate the risk to reduce as much as possible the impact of third party slowness, this can be done: by rules on WAF, package modifications directly on the machines (even if I’m not a fan of this solution), change the exposure of some machines (move machines behind a VPN for example) or interrupt some services temporarily.

The temporary solution depends, of course, on the assessment of the associated risk.

How I lived this stage

As I mentioned, I was a bit frustrated by the lack of response from some teams, or the impression that they didn’t care… For some teams, I’m just one incident among many already in progress, and I’m just adding a piece to the machine.

This step is not necessarily simple, but I find it simpler than having the inventory.

After log4j

For the moment the log4j storm is not over yet, with a flaw discovered the day before yesterday as I write these lines.

Invest in security

This storm, like others before it (Spectre, Meltdown, Heartbleed, etc.) is an opportunity for security teams to remind the importance of certain investments:

• Having a security team that keeps up to date
• Having the tools to quickly identify vulnerabilities
• Having an up-to-date inventory
• Educate teams on the importance of good security practices

I even reiterated this point (on Twitter once again, in French) recently:

Fighting the predatory behavior of big tech

This flaw reminds us of the harsh realities of open source projects: many projects are clearly exploited by many huge companies that rely on them without ever contributing either code or at least financially.

Log4j is a library exploited by millions of applications in the world, yet it is now by few people.

Many techs in the open source world have made the same observation (in French, once again):

This is not unlike recent battles:

• Elastic.co VS AWS
• MongoDB VS AWS

In the absence of a viable solution, the solution is often to go through the legal process by putting clauses preventing the use of these companies, because they exploit the wealth of these projects without ever giving anything in return.

Some time ago, I talked to you about tracking and why I care about my privacy. In my conclusion, I indicated that user tracking was still a useful tool for a company, as long as it was ethical and respectful of users.

However, very often, I see that Google Analytics is used by the sites I browse on. It is far (even very far) from being respectful of your users' data. Even worse! You allow Google to know the activity of your site from end to end and to know how to better target its ads (among others).

Today, I propose you see how it is possible to track users while showing respect for them.

What is Matomo?

Matomo is an OpenSource alternative to Google Analytics. Available on GitHub, it allows you to deploy a complete application with a dashboard and a tracking system in JavaScript. It is also possible to rely on reading server logs rather than JavaScript.

GitHub - matomo-org/matomo: Liberating Web Analytics. Star us on Github? +1. Matomo is the leading open alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. We love Pull Requests!

Liberating Web Analytics. Star us on Github? +1. Matomo is the leading open alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from webs...

GitHubmatomo-org

Having an open source solution allows you to know its intricacies, but above all and the guarantee of respect for your users. The free software carries today the majority of the internet (including this web site).

Matomo allows you to have the statistics classically used as (in a non-exhaustive way):

• The visited pages
• The "User Agent" (OS, Browser, language, etc.)
• The transformation rate of social networks
• The GeoIp
• The bounce rate
• The duration of sessions

Based on the classic, but efficient PHP + MySql couple, the application is light and also integrates components aimed at respecting privacy:

• Support of the DoNotTrack header, which allows you to indicate that you do not want the website to track your activity
• Support for RGPD constraints (informed consent), it is possible to activate or deactivate Matomo directly for each user
• Native anonymization or pseudoanonymization of data

Thus it is possible to have clear and complete statistics without being intrusive.

Note that there is also a solution managed directly by Matomo, available on their website.

Matomo - L’alternative à Google Analytics qui protège vos données

Matomo est l’alternative éthique à Google Analytics qui protège vos données et la vie privée de vos clients Une puissante plateforme d’analyse Web avec la propriété de 100 % des données.

Analytics Platform - MatomoHimanshu Sharma | Digital Marketing Consultant

In the case I'm going to present today, we'll start from the official docker image.

Deploying Matomo in Kubernetes

As usual on this blog, I'm going to talk about deploying in Kubernetes, because that's what I have on hand and it's becoming more and more common today.

As a reminder, my deployment is not a "prod ready" deployment in enterprise, it doesn't handle high availability and scaling is catastrophic since the database is in the same pod. This post is simply a "demo" of what is possible so you can easily adapt it to your needs. The idea is, of course, that you can make the tool your own.

Although the deployment is in Kubernetes, it remains easily transposable to Docker or Docker-compose (or podman, or rancher, etc.).

So here's the deployment we're going to do, in blue are the elements we're going to create:

So we will have 3 yaml files:

• A "deployment" carrying Matomo and MySql
• A "service" allowing exposing Matomo
• IngressRoute allowing Traefik to serve Matomo and manage its TLS certificate

The deployment

We are here on something very basic:

• Two images (Matomo and MySql)
• A database configuration (user, default database, etc.)
• Some persistent storage

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: matomo
  labels:
    app: matomo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: matomo
  template:
    metadata:
      labels:
        app: matomo
    spec:
      containers:
      - name: matomo
        image: matomo:4.1.1-apache
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        env:
        - name: MATOMO_DATABASE_HOST
          value: 127.0.0.1
        - name: MATOMO_DATABASE_USERNAME
          value: "root"
        - name: MATOMO_DATABASE_PASSWORD
          value: "matomo"
        - name: MATOMO_DATABASE_DBNAME
          value: "matomo"
        volumeMounts:
        - name: matomo-storage
          mountPath: /var/www/html/config
      - name: mysql
        volumeMounts:
        - name : mysql-storage
          mountPath: /var/lib/mysql
        image: mysql:8.0.23
        imagePullPolicy: Always
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "matomo"
        - name: MYSQL_DATABASE
          value: "matomo"
        resources:
          limits:
            cpu: "0.2"
            memory: "512Mi"
      volumes:
      - name: mysql-storage
        hostPath:
          path: /home/kube/matomo/content/mysql
          type: Directory
      - name: matomo-storage
        hostPath:
          path: /home/kube/matomo/content/matomo
          type: Directory

We find, first of all, the Matomo container, based on an Apache image.

This last one has as configuration :

• A binding on port 80 of the pod
• The configuration of the database. In my example, I don't need to put a strong password, because the database is dedicated to Matomo and is only accessible by the latter.
• A persistent storage. This allows Matomo to store its configurations locally (especially the database configuration). This configuration is not valid for a high availability deployment, as I use local storage

Then, we have a second container that carries the MySql database, with the following configuration:

• Logins and passwords on the database
• The binding of port 3306 inside the pod only, so only accessible by Matomo
• A persistent storage : I don't want to lose my data at the slightest reboot. Once again, this configuration is not valid for a high availability deployment.

Then, we have a service, which will allow us to expose Matomo :

kind: Service
apiVersion: v1
metadata:
  labels:
    app: matomo
  name: matomo
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: http
  selector:
    app: matomo

Nothing transcendent here, I simply bind the port 80 of my pod to a Kubernetes ClusterIP.

Finally, I declare IngressRoute, this is the one that will allow Traefik to route traffic to my pod.

kind: IngressRoute
metadata:
  name: matomo-tls
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - kind: Rule
    match: Host(`matomo.tferdinand.net`)
    services:
    - name: matomo
      port: 80
    middlewares:
      - name: security
  tls:
    certResolver: le
    options:
      name: mytlsoption
      namespace: default
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: matomo
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    match: Host(`matomo.tferdinand.net`)
    services:
    - name: matomo
      port: 80
    middlewares:
      - name: security
      - name: redirectscheme

We find here 2 blocks, one in HTTP and one in https, with an automatic redirection from one to the other.

Then, we can see that I assign the DNS record "matomo.tferdinand.net" to this one.

Finally, I apply the same security patterns that I had described in one of my previous posts.

I can now apply my 3 configurations.

Configuring Matomo

Configuring the server

By connecting to the address declared in Traefik, you should see the installation wizard.

It will guide you step by step to :

• Configure your database
• Configure your first administrator user
• Configure your first site

You can see all the steps below:

Integrate the js tracker

As you could see, Matomo works via a JavaScript tracker to be integrated in your website.

It is this last one which will interact with the client to give the necessary information.

To activate Matomo on my site, I just have to integrate this code. In the case of Ghost, my blog, I have a parameter for that in "Code injection" > "Site headers".

When you connect to the Matomo interface, you should see some traffic, taking into account that :

• Browsers sending DNT (Do Not Track) header will not send any information (like Brave for example)
• From an RGPD perspective, you must obtain the user's consent before collecting their information and placing a cookie on their computer. Matomo provides a guide to this (https://matomo.org/docs/gdpr/).

To go further

It is also possible, as I mentioned above, to couple Matomo to log files.

There is indeed a python script delivered in the Matomo image which allows you to do this action, starting from known log formats, but also to define custom formats.

To use this script, there are two prerequisites:

• Python 3.x
• PHP 7.x

However, the image used above is indeed based on the official PHP image, as can be seen on the GitHub repository, but does not have Python installed.

docker/Dockerfile-debian.template at master · matomo-org/docker

Official Docker project for Matomo Analytics. Contribute to matomo-org/docker development by creating an account on GitHub.

GitHubmatomo-org

If you want to use this script, there are two solutions:

• Create a custom image for this script with Python and PHP
• Start from the official image by adding PHP, like this

FROM matomo:4.1.1-apache
RUN apt update -y 
RUN apt install -y python3
RUN apt-get purge -y --auto-remove && rm -rf /var/lib/apt/lists/*

The script is then easily usable, as indicated in the official documentation :

How to use Log Analytics tool - Analytics Platform - Matomo

Analytics Platform - Matomo

Regarding the authentication, three solutions are possible. Either you are in the image that runs the application, and the script will be able to exploit directly the configuration files and thus connect directly.

The second solution is to create a token which will allow you to authenticate yourself and to pass it in the parameter with the switch --token-auth, this token can be created from the interface of Matomo. To do so, you just have to click on the cog in the top right corner, then "Security" and "Create a new token".

Be careful, the token can only be recovered once, so don't forget to store it in a secure environment.

Finally, the third solution, which I strongly advise against, it is also possible to pass directly a login and password in command line.

Importing logs with a Kubernetes cronjob

For my part, I chose to import the logs by using a Kubernetes cronjob.

Indeed, I chose not to deploy the javascript that I find too intrusive for you, my visitors. So I chose to analyze the raw logs provided by Traefik.

I also chose not to use the official image for this purpose and I use a token authentication.

The image

The docker image I used is quite basic:

FROM python:3.8.7-buster
RUN apt update\
	&& apt install git \
	&& git clone https://github.com/matomo-org/matomo-log-analytics.git \
	&& rm -rf /var/lib/apt/lists/*

I start from a python 3.8 image (because the importer is only compatible with some versions) then I clone the associated repository.

You can get the image directly created via dockerhub :

Docker Hub

The job
I chose to put my Traefik logs in a directory that I can mount when I need it.

To know more about logs configuration, I invite you to read my article on this subject :

Extraction of Traefik accesslogs and dashboard creation

A few weeks ago, I wrote an article explaining the migration from Traefik 1 to Traefik 2, but this time I propose to address a crucial point in the implementation of an application, its monitoring. This article explains how I set up my dashboarding, it doesn’t explain in any case

TFerdinand.netTeddy FERDINAND

I then created a Kubernetes cronjob, which will allow me to read the logs at regular intervals to integrate them into Matomo.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: matomo-importer
spec:
  schedule: "5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: matomo-importer
            image: tferdinand/matomo-log-importer:1.0.0
            imagePullPolicy: IfNotPresent
            env:
              - name: PROCESSING_DATE
                value: $(date -d "1 hour ago" "+%d/%b/%Y:%H")
              - name: TOKEN
                value: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
              - name: TRAEFIK_INGRESS_ID
                value: "default-traefik-web-ui-tls-c463f039d72c55f0aca6"
              - name: ID_SITE
                value: 1
            command:
            - /bin/bash
            - -c
            - grep $(TRAEFIK_INGRESS_ID) /tmp/access.log | grep $(PROCESSING_DATE) > /tmp/matomo_$(ID_SITE).log; python matomo-log-analytics/import_logs.py  --url=https://matomo.tferdinand.net --idsite=$(ID_SITE) --debug --token-auth $(TOKEN) --log-format-name=common /tmp/matomo_$(ID_SITE).log
            volumeMounts:
            - name: shared-storage
              mountPath: /tmp/
          restartPolicy: OnFailure
          volumes:
            - name: shared-storage
              hostPath:
                path: /home/kube/share/from_traefik
                type: Directory

So I run this task once an hour, to process the previous hour.

The "grep" that we see at the beginning allows me to output only the logs for the service I want, the Matomo log importer not knowing how to handle the format of the traefik logs in a more refined way.

ID_SITE allows me to indicate the site ID on Matomo, by default 1.

So, every hour, I can follow the site's traffic, without impacting my users. Note that it is possible to go down to the minute by adjusting the parameters.

In conclusion

Matomo is a reliable alternative that is more respectful of your users. It is completely possible to consider it in a professional environment.

If you have a need for more consistent statistics on your site, it is inherently better for your users to use this solution than Google Analytics.

Moreover, hosting the solution yourself also allows you to respect the constraints more easily related to the RGPD. The openness of the code also guarantees that no third party exploitation of the data is made.

I also insist on the fact that not all sites need complete tracking, for my part, a simple exploitation of the logs allows me to get consistent data.

Because of my professional background, I’ve worked for a lot of companies, either as a service provider or internally.

Salary is often:

• A taboo subject, you should not talk about the salary you receive and even less with your colleagues
• Seen as the only criterion for recruiting: If we don’t recruit, it’s because we don’t pay enough, if employees leave, it’s because of the salary

Today, I give you my vision on this point.

The salary is an important criterion…

Like many I think, I work to earn money. To say that salary is not a criterion for choosing a company would be hypocritical on my part.

Clearly, the salary that a company will offer me is one of the points I look at. However, for me it is only one of the criteria.

In the past, I had the opportunity to work in many different sectors and sizes of companies:

• retail: Carrefour, Fnac, Conforama
• industry: L’Oréal
• banking: ING Commercial Banking, Société Générale
• Media: Médiamétrie, Groupe SeLoger

All this to say that beside the salary, this panel also allowed me to see the sectors that interest me and what is not for me.

It is also thanks to this that I understood that I prefer to work in smaller structures than in big multinationals.

In IT, concerning the salary, you have to keep in mind that you often get a better salary in SSII/ESN. However, you have to keep in mind that this is not necessarily adapted to everyone, the world of service provision is a world that is more “mobile”, where you change context regularly, it is not necessarily adapted to everyone.

… but far from being the only one

It is indeed important to be paid according to the work done and the quality of it, that is undeniable.

However, limiting the choice of a company to the salary it offers is simplistic.

The general atmosphere

Being in a company with a healthy general atmosphere is just as important. I’m not talking about table soccer and croissants, but about a healthy atmosphere for your mind.

As an anecdote, in one of my former missions, I was shocked by a situation: I was on the client’s set surrounded by internal teams, and some people could not communicate other than by insults or aggression, I even saw one person crying on the set! Atmosphere!

This allowed me to see that this is not what I am looking for, and to tell myself that being in a healthy work environment is essential for me.

Feeling valued

I am one of those people who believe that in technical positions, it is important that the work you do is valued. We are often “men (and women, of course) behind the scenes”, and when a project is successful, the technical teams that made it possible are rarely put forward.

There are several ways to do this for me:

• The salary increase (this seems logical)
• Saying it! (we often forget this detail)
• Share the success, for example by doing communications to explain what the team did
• Highlight it on a technical blog

The last point is important to me. More and more companies are setting up technical blogs, even if it’s not their core business. Why? Because it allows them to highlight the skills they cultivate internally, but not only, a technical blog is also a perfect showcase for recruiting. Be careful, though, it is important that the author is highlighted in these articles, and not necessarily the company (even though the goal is visibility for the company)

At WeScale, my current employer, this also manifests itself with the WeTribute, once a month, we receive a form through which we can thank several colleagues. Once a quarter, all these thanks are compiled and sent and the three most thanked (excluding management) share a bonus. Being recognized by your peers is important and I must admit that receiving these messages once a quarter is always heartwarming!

Knowing where you’re going

The reason I left some companies was the captainless ship syndrome. You know, that feeling that you’re sailing with the waves, but no one has set a clear course.

From my point of view, it’s a frustrating situation. I like to know where the company I’m in is going. This requires communication: explaining the company’s choices, sharing a roadmap (even a very “big picture” one)… Basically, having a clearly defined direction.

But it also involves a point that is often forgotten: the career plan. I’m one of those people who need to be able to project themselves and say “OK, if I stay x years in the company, how can I evolve? Personally, I need to know that I won’t be doing the same thing every day until I quit.

Having visibility on possible evolutions is something that is as important as the salary.

You are very kind, but how do I recruit?

I’m not an expert in human resources, far from it, it’s a job that requires a lot of psychology and empathy and I don’t intend to take a position on these subjects.

However, I am one of those who are on the other side, and I can give my point of view.

First of all, stop putting forward your table soccer, resting room and co., I am applying for a job, not a summer camp!

Then, don’t only focus on the salary, but on what’s around it. I have turned down offers with a 50% increase in salary because the work environment was not for me. Paying a lot of money to people who won’t stay because of the work environment actually costs you more (training time, continuous recruitment, team turnover = demotivating, etc.)!

Some levers like telecommuting for example are nowadays points that can also be interesting.

Today, it is important to have healthy working conditions to be able to recruit. Don’t forget, for the same salary, candidates will always go where they feel most comfortable.

I’ve been working in the IT field for more than 10 years now and I’ve worked with a lot of “security” teams within the companies I’ve been in. I’ve been a security guy (Cloud Security Architect) for a little over a year now.

During these years, I often noticed a blocking posture of the security teams, sometimes even disconnected from the field, leading to slowdowns and tensions in the projects.

Today, I’m going to present you the posture I chose to adopt and explain why. I will also give you my point of view with a little hindsight after one year.

A little background

In most of the companies I’ve worked for, security is often a mandatory checkpoint, but not necessarily useful or efficient… I’ll explain.

Often, this team is called upon at the end of the chain to:

• Perform a penetration test on an infrastructure
• Verify that the deliverable is compliant
• Ask for specifics “right passes”

Moreover, we avoid soliciting them as much as possible, because we know that the response time will be long, often in weeks.

To add another layer, I have often had to deal with people who are disconnected from the reality of the field concerning:

• the maturity of the implementation teams concerning cybersecurity
• tooling
• business needs
• delivery times

In other words, we often end up with a deaf language, each one not understanding the other.

When I arrived in my mission, I was in a position where a barrier existed between the dev/ops and security. The former did not understand the interest of the measures being implemented.

This barrier creates resistance to change and therefore slows down the speed of the teams, in addition to creating a not necessarily pleasant atmosphere.

Coming down from the ivory tower

One of the major issues, as I mentioned earlier, is the disconnect between the operations (DevOps) teams and security.

One of the first things I manage to deploy to improve this situation was to organize regular check-ins between myself and the ops.

In this model, the ops team is an “ambassador” for security. This doesn’t mean that I don’t exchange with the development teams, but rather that a SPOC (Single Point Of Contact) must be designated. Humanly speaking, I can’t talk directly to 200 people.

Indeed, the purpose of this point is mainly to be bidirectional. This allows:

• Security can present its roadmap in advance of the phase, by explaining it
• That the teams can bring up their concerns, needs or current blockages as soon as possible
• Generally speaking, this improves communication between entities

Moreover, taking into account the feedback from the “field” teams also allows adjusting as much as possible the tools or the security rules that are being implemented.

Be careful, however, the goal is not to justify the choices, but to explain them. You must therefore be in a pedagogical posture, and not be defensive or defensive.

Zero trust … yes, but not only!

I am a fan of the zero trust model (a blog post on the subject is coming soon); however, I am among those who consider it a model, a target to reach, not an immediate objective.

The concern I have often seen is an inconsistent posture on this subject:

• Hardening too much: we prevent teams from working by putting too strong a technical framework (for example, by not giving any autonomy on a software installation)
• Not being able to adapt a rule if it is too rigid (e.g., network filtering, or an incompatible protocol)
• Being in a blocking mode on a project, preventing any delivery

My approach is more to say that we need to understand the technical and operational constraints of the teams to see how we can best apply zero trust.

This means:

• That I sometimes tolerate giving more rights than necessary temporarily to allow projects to progress
• That it is necessary to trust the teams
• That we must show that we are available to answer questions, and that we are in a constructive approach, the goal is to find a solution

Supporting projects rather than blocking them

This is the point I often insist on. In my mind, security is meant to accompany projects as best as possible. The idea is therefore :

• To document upstream as much as possible, so that the information is shared
• To be available to help the teams
• To listen to the needs of the teams
• To have a benevolent approach
• To be in an advisory role rather than a “finished work inspector

You will have understood, I am completely in a DevSecOps approach, which, from my point of view, allows to be efficient.

Positioning oneself as a coach also saves time, by becoming a full-fledged project actor, it allows to be listened to and to ensure that security needs are taken into account as early as possible in the projects.

Adopting this posture also allows the teams to become allies rather than adversaries to be fought.

This also requires a lot of communication, as I often say, a large part of my daily work is to explain and document. The clearer and more available information is, the more the associated rules will be adopted.

This also means that a CISO must know how to surround himself with technical people in order to best support the teams. As a reminder, the role of CISO is not so much technical as organizational.

In conclusion

I apply to myself the rules I mentioned earlier, and I must admit that in one year, I have seen the direct benefits:

• I am asked at the beginning of the next ones to see what points of attention I can bring
• Security decisions are understood (not necessarily accepted, but the objective is understood)
• I am no longer seen as a blocking point

My approach probably comes from my background as an Ops person, which means that I myself have been on the other side of the fence. I am therefore better able to understand the problems that teams in the field may face.

This does not mean that everything is rosy, there will always be dissatisfied people, but instead of being involved at the end of the chain, I have become an actor in the projects.

Confined spaces have changed our work habits a lot. Telecommuting has become something more common than it was just a year ago.

With the implementation of telecommuting very quickly, new risks have appeared. Today, I suggest talking about Grey IT.

What is Grey IT?

In a company, in a classical way, the applications used are referenced in a service catalog.

For example, if your company uses Slack, the office service knows it, and will configure this application so that it works with the company’s security and confidentiality standards.

But before the app is even installed, it is first reviewed:

• By legal: The goal here is to check the terms of use, to see if any usage could be detrimental to the company, such as the fact that the information put into the app would become public or no longer belong to your company
• By security: The goal here is to control the compliance of the application with the security standards of your IS. It also means analyzing the code if necessary or launching analyses (malware, known CVEs, etc.).
• Accounting: An application always has a cost, visible or not (I will come back to this point), and it is important to see if this cost is acceptable in relation to your budget, or if the need is not already covered by another application that you already have

Grey IT is precisely the fact of not following these processes often perceived as restrictive.

However, their purpose is to secure your business, but from the outside, it can be seen as heavy and slow. Typically, if we take my example of slack, I may have a need that slack does not meet today, like having a whiteboard.

So, to answer my need, I create a personal Gmail account to use Google Meet and share whiteboards with my colleagues. Don’t worry, it’s free!

If it’s free, it means you are the product

My previous example is the case I’ve often come across: I have a need, a free product meets it, I take the product.

But here’s the thing: developing professional applications cannot be improvised, and although there are obviously open source solutions hosted by these same communities, this remains on the fringe of many other applications that have the means to be more visible.

To take Gmail again, Google is not a non-profit organization. Their business is your data, your usage, your habits. It’s how Google makes money by offering you a free service.

For some it will be a “freemium” model, like Slack which has a free offer, but which quickly shows its limits in business.

That’s why you have to study the business model and the contractual part carefully before choosing an application, because it can sometimes backfire. Providing data about your business to a third party company can :

• Be dangerous (for example in terms of GDPR data)
• Allow this company to exploit this idea for you
• Make this company grow on your workload

Why is this a danger?

Using non-referenced applications poses several concerns:

• Creation of parallel IS
• Adding security risks with “exotic” applications
• Addition of confidentiality risks
• Increasing the overall budget, as several entities may purchase applications with overlapping needs, but on a smaller scale, not benefiting from volume pricing. It also means potentially having multiple applications that meet the same need.

I can’t count the number of times I’ve been shown the newest trendy application and upon digging deeper I’ve realized that it’s trendy but not at all usable in the current business context.

The most obvious example, in my opinion, is the case of Zoom. This application was a huge success at the beginning of the first containment, because it met real functional needs.

However, in terms of security, it was a disaster:

• No end-to-end encryption
• Zoom bombing: the invasion of conferences by outsiders who can at best spoil your meeting, at worst remain discreet and do industrial espionage
• Security concerns on user workstations that allow for access rights

We talked about this last June in the WeScale podcast [FR Link]!

Thus, following a normal company process, it would have been simply refused in many companies (besides Zoom is still banned in many companies); nevertheless, it has been used a lot, because it was exploited without following these processes in order to respond faster to a need.

To conclude

Having new needs is normal in a company. However it is important to follow well-defined processes when adding a new application to your IS.

This is to secure your business and avoid simply losing value.

Also, it is important to have an up-to-date inventory of the applications used and available to avoid adding applications when the needs are already covered. Often, the Grey IT cases I saw simply came from there.

A few months ago, I wrote a post to explain how to easily create a local Kubernetes cluster leveraging Vagrant and Traefik. You can find it here:

Create a local Kubernetes cluster with Vagrant

Testing Kubernetes is quite easy thanks to solutions such as Minikube. However, when you want to test cluster-specific features, such as load balancing or failover, it is not necessarily suitable anymore. It is possible to build your Kubernetes infrastructure on servers, or by using managed services…

TFerdinand.netTeddy FERDINAND

Today, I suggest to see how we can accelerate this creation by building ourselves the box used by Vagrant, preconfigured with our tools. This post is the continuation of the one above. Some notions will not be discussed again.

What is Packer?

Packer is also a HashiCorp tool, like Vagrant. Its role is to pack (hence its name) virtual machines.

It allows you to create AWS AMIs, Docker images, VirtualBox virtual machines and so on. You can find the complete list of providers managed by Packer here.

The main strengths of Packer are :

• The simplicity of the configuration: a simple JSON or HCL file can describe the desired build
• Parallelization: you can create the same image on several providers in parallel, very useful in a multicloud approach
• Reproducibility: It is easy to recreate an OS image from scratch, using only the Packer files, which allows sharing only these files, rather than large images

Today, we will create a Vagrant box starting from the basic Ubuntu server ISO.

Why use Packer to create Vagrant boxes?

If we go back to my previous post, we had seen how to quickly start a K3S cluster with Vagrant. However, on each virtual machine :

• I had to download K3S or Traefik
• I had to do my SSH configuration (for the trust between nodes)
• I couldn't make sure that the version of K3S and the set of binaries was exactly the same from one run to another. (I'll come back to this point)

So, to solve this problem, I will be able to use Packer to create boxes directly configured to meet my needs.

This will allow me to start my cluster more quickly and not to have to download and configure everything at each launch.

What are the steps to create the box?

First, we will create a configuration file for Packer. This file can be written in JSON or in HCL (since recently), the language specific to HashiCorp tools. For this project I arbitrarily chose HCL. Note that JSON can perform exactly the same actions. This file will allow us to indicate the basic ISO that we will use, as well as all the necessary actions until the creation of the box.

Then we will provide this configuration file as input to Packer which will do the following operations:

• Download the image of the requested OS
• Check the checksum of the image
• Create a virtual machine to launch the image
• Execute the installation commands
• Execute the machine provisioning commands
• Shut down the VM
• Export as Vagrant box
• Delete the temporary VM

These operations will be done in a completely transparent way, on our side, we will only run one command.

Hands in the mud !

As a preamble, you can find all the scripts described below on the associated repository :

GitHub - teddy-ferdinand/packer-vagrant-k3s-cluster

Contribute to teddy-ferdinand/packer-vagrant-k3s-cluster development by creating an account on GitHub.

GitHubteddy-ferdinand

First, you will need Packer and Vagrant. For Vagrant, I invite you to look at my previous post to know how to install it. Concerning Packer, I invite you to look at the dedicated page on the editor's website in order to have the most adapted installation to your operating system.

To make my box, I chose to have three files :

├── http
│   └── preseed.cfg
├── packer_installer.sh
└── packer.pkr.hcl

So we have :

• An http directory: this directory will be exposed as a web server by Packer to be able to directly operate a network installation to preconfigure the OS. I will come back to this point in the configuration.
• A packer_installer.sh script : this script contains the basic provisioning of the image, e.g., the installation of K3S, traefik and the deployment of my SSH keys
• A configuration file packer.pkr.hcl : This file is the configuration file that we will use with Packer to create our box. Warning: the .pkr.hcl extension is necessary for Packer to detect the right format; otherwise it will try to process it as JSON.

The Packer configuration

Let's start with the Packer configuration file. This is the file that will define our box.

First of all, we have the first section: the source. As its name indicates, this element allows us to indicate which image we have to start from for the build, as well as the basic configuration (login, commands to be executed at boot time, etc.).

source "virtualbox-iso" "ubuntu" {
  guest_os_type = "Ubuntu_64"
  iso_url = "http://cdimage.ubuntu.com/ubuntu/releases/bionic/release/ubuntu-18.04.5-server-amd64.iso"
  iso_checksum = "sha256:8c5fc24894394035402f66f3824beb7234b757dd2b5531379cb310cedfdf0996"
  ssh_username = "packer"
  ssh_password = "packer"
  ssh_port= 22
  shutdown_command = "echo 'packer' | sudo -S shutdown -P now"
  http_directory = "http"
  guest_additions_mode = "disable"
  boot_command = [
        "",
        "",
        "",
        "/install/vmlinuz",
        " auto",
        " console-setup/ask_detect=false",
        " console-setup/layoutcode=us",
        " console-setup/modelcode=pc105",
        " debconf/frontend=noninteractive",
        " debian-installer=en_US",
        " fb=false",
        " initrd=/install/initrd.gz",
        " kbd-chooser/method=us",
        " keyboard-configuration/layout=USA",
        " keyboard-configuration/variant=USA",
        " locale=en_US",
        " netcfg/get_domain=vm",
        " netcfg/get_hostname=packer",
        " grub-installer/bootdev=/dev/sda",
        " noapic",
        " preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg",
        " -- ",
        ""
      ]
}

In this part, we find :

• The base image: Ubuntu 18.04.5
• The checksum of the image, this will allow Packer to control the integrity of the image after its download
• The logins and passwords to use in SSH (which will be used in the second part)
  Which command should be run to shut down the machine properly
• The http directory, which allows exposing a preseed file that carries the basic configuration of my image. You can find it on line 32.
• The fact that I disable the installation of the guests additions. The guests additions allow a better integration between the host and the VM, in our case, I don't need them, so I might as well not install them.
• Finally, we find the commands launched at boot time. These commands will be launched in "emulation" mode. This means that Packer will emulate a keyboard to type these commands in your virtual machine.

Concerning the commands launched at build time, I was inspired by a repository containing basic Packer configurations:

GitHub - geerlingguy/packer-boxes: Jeff Geerling’s Packer build configurations for Vagrant boxes.

Jeff Geerling’s Packer build configurations for Vagrant boxes. - GitHub - geerlingguy/packer-boxes: Jeff Geerling’s Packer build configurations for Vagrant boxes.

GitHubgeerlingguy

Still in this same configuration file, we have a second block, the "builder". As its name indicates, the role of this block is to build our box. Let's see its content.

build {
    sources = ["source.virtualbox-iso.ubuntu"]

    provisioner "file"{
        sources = [".ssh/id_rsa", ".ssh/id_rsa.pub"]
        destination = "/tmp/"
    }

    provisioner "shell"{
        script = "./packer_installer.sh"
        execute_command = "echo 'packer' | sudo -S -E sh -c '{{ .Vars }} {{ .Path }}'"
    }

    post-processor "vagrant" {
        keep_input_artifact = false
        provider_override = "virtualbox"
    }
}

Those who are used to HCL will see that the source is a direct reference. This means that Packer natively solves a dependency link between the two elements.

Then we can see that I use "provisionners," the role of these elements is precisely to do the provisioning, there are dozens. It is for example possible to use Ansible to do its configuration.

You can find the complete list of provisioners here: https://www.packer.io/docs/provisioners.

For my part, I chose to use two of them:

• I copy my SSH keys that I have generated in my virtual machine
• I then run an installation script that I made, note the invocation of sudo for the launch. By default Packer runs the commands via the user specified in sources.

Finally, I have a "post-processor", the role of this element is to export my box once the configuration of my virtual machine is finished. In my case, the export is Vagrant, which means that packer will create a box vagrant from the Virtualbox virtual machine. As for providers and provisioners, there are many post-processors: https://www.packer.io/docs/post-processors

The installation script

As we have seen above, I have a small Shell script that is invoked to perform the provisioning of my machine. Here is the content of this script, which is quite basic.

#!/bin/sh
# Deploy keys to allow all nodes to connect each others as root
mkdir /root/.ssh/
mv /tmp/id_rsa*  /root/.ssh/

chmod 400 /root/.ssh/id_rsa*
chown root:root  /root/.ssh/id_rsa*

cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
chmod 400 /root/.ssh/authorized_keys
chown root:root /root/.ssh/authorized_keys

# Install updates and curl
apt update
apt install -y curl
# Apt cleanup.
apt autoremove -y
apt update

#  Blank netplan machine-id (DUID) so machines get unique ID generated on boot.
truncate -s 0 /etc/machine-id
rm /var/lib/dbus/machine-id
ln -s /etc/machine-id /var/lib/dbus/machine-id

# Download k3s
curl -L https://get.k3s.io -o /home/packer/k3s
chmod +x /home/packer/k3s

# Download Traefik
curl -L https://github.com/containous/traefik/releases/download/v2.2.11/traefik_v2.2.11_linux_amd64.tar.gz -o /home/packer/traefik.tar.gz
cd /home/packer
tar xvfz ./traefik.tar.gz
rm ./traefik.tar.gz
chmod +x /home/packer/traefik

At first, we can find the deployment of my SSH key. When we were running Vagrant, it was him who did this installation. Now Packer does it for us. This will also allow us to connect directly using these keys.

Then I update my packages and install curl, because the image I use is a "minimal" image without this package. Then I clean up the useless packages. Besides the aspect of a clean installation, it allows reducing the size of my box.

A small cleaning is then done so that our machine does not have an identifier linked to it, a very important point for Vagrant, without which we can have conflicts between our virtual machines.

Finally, I download K3S and Traefik, which I put in /home/packer. This will allow Packer to use them directly.

The preseed file

This file is probably the most "raw", but very important, because it is the one that will perform the installation of the base of our machine.

d-i base-installer/kernel/override-image string linux-server
d-i clock-setup/utc boolean true
d-i clock-setup/utc-auto boolean true
d-i finish-install/reboot_in_progress note
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i partman-auto/disk string /dev/sda
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic
d-i partman-auto/method string lvm
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-lvm/device_remove_lvm boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman/confirm_write_new_label boolean true
d-i pkgsel/include string openssh-server cryptsetup build-essential libssl-dev libreadline-dev zlib1g-dev linux-source dkms nfs-common
d-i pkgsel/install-language-support boolean false
d-i pkgsel/update-policy select none
d-i pkgsel/upgrade select full-upgrade
d-i time/zone string UTC
tasksel tasksel/first multiselect standard, ubuntu-server

d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/layoutcode string us
d-i keyboard-configuration/modelcode string pc105
d-i debian-installer/locale string en_US.UTF-8

# Create packer user account.
d-i passwd/user-fullname string packer
d-i passwd/username string packer
d-i passwd/user-password password packer
d-i passwd/user-password-again password packer
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false
d-i passwd/user-default-groups packer sudo
d-i passwd/user-uid string 900

These include, but are not limited to:

• The configuration of the language and the keyboard layout
• The partitioning of my disks
• The basic packages to install
• The creation of my user "packer

Let's launch it all!

Once we have all our files, we can launch the construction of my Vagrant box with packer with a simple :

packer build packer.pkr.hcl

As you can see below, this takes a few minutes and will perform all the actions I mentioned earlier.

Once it is finished, we have a box "packer_ubuntu_virtualbox.box" in our current directory. During the execution, we can see the Virtualbox window, this is a choice I made. It is possible to disable its display by adding

headless = true

in my source.

Using this box in Vagrant

Now that my box is created, I can use it in Vagrant. The basic scripts are very similar to the original ones, except that :

• I don't start from a public Ubuntu box anymore, but from my local box
• I don't need to deploy my RSA keys anymore
• I use the connection via the RSA key instead of the native vagrant login
• I don't need to download K3S and Traefik anymore
• I disable the synchronization of the Vagrant directory that I don't use, since it requires the use of the "guests additions" that I haven't installed

So here is the file in question :

MASTER_COUNT = 3
NODE_COUNT = 3
IMAGE = "packer_ubuntu_virtualbox.box"

Vagrant.configure("2") do |config|

  (1..MASTER_COUNT).each do |i|
    config.vm.define "kubemaster#{i}" do |kubemasters|
      kubemasters.vm.box = IMAGE
      kubemasters.vm.hostname = "kubemaster#{i}"
      kubemasters.vm.network  :private_network, ip: "10.0.0.#{i+10}"
      kubemasters.vm.provision "shell", path: "scripts/master_install.sh"
      kubemasters.ssh.username = "root"
      kubemasters.ssh.private_key_path = ".ssh/id_rsa"
      kubemasters.vm.synced_folder '.', '/vagrant', disabled: true
    end
  end

  (1..NODE_COUNT).each do |i|
    config.vm.define "kubenode#{i}" do |kubenodes|
      kubenodes.vm.box = IMAGE
      kubenodes.vm.hostname = "kubenode#{i}"
      kubenodes.vm.network  :private_network, ip: "10.0.0.#{i+20}"
      kubenodes.vm.provision "shell", path: "scripts/node_install.sh"
      kubenodes.ssh.username = "root"
      kubenodes.ssh.private_key_path = ".ssh/id_rsa"
      kubenodes.vm.synced_folder '.', '/vagrant', disabled: true
    end
  end

  config.vm.define "front_lb" do |traefik|
      traefik.vm.box = IMAGE
      traefik.vm.hostname = "traefik"
      traefik.vm.network  :private_network, ip: "10.0.0.30"   
      traefik.vm.provision "file", source: "./scripts/traefik/dynamic_conf.toml", destination: "/tmp/traefikconf/dynamic_conf.toml"
      traefik.vm.provision "file", source: "./scripts/traefik/static_conf.toml", destination: "/tmp/traefikconf/static_conf.toml"
      traefik.vm.provision "shell", path: "scripts/lb_install.sh"
      traefik.vm.network "forwarded_port", guest: 6443, host: 6443
      traefik.ssh.username = "root"
      traefik.ssh.private_key_path = ".ssh/id_rsa"
      traefik.vm.synced_folder '.', '/vagrant', disabled: true
  end
end

As you can see, no huge changes here from my original script.

The real changes are in the Vagrant deployment scripts.

K3S installation

Installing the master

#!/bin/sh

# Add current node in  /etc/hosts
echo "127.0.1.1 $(hostname)" >> /etc/hosts

# Get current IP adress to launch k3S
current_ip=$(/sbin/ip -o -4 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1)

# If we are on first node, launch k3s with cluster-init, else we join the existing cluster
if [ $(hostname) = "kubemaster1" ]
then
    export INSTALL_K3S_EXEC="server --cluster-init --tls-san $(hostname) --bind-address=${current_ip} --advertise-address=${current_ip} --node-ip=${current_ip} --no-deploy=traefik"
    export INSTALL_K3S_VERSION="v1.16.15+k3s1"
    sh /home/packer/k3s
else
    echo "10.0.0.11  kubemaster1" >> /etc/hosts
    scp -o StrictHostKeyChecking=no root@kubemaster1:/var/lib/rancher/k3s/server/token /tmp/token
    export INSTALL_K3S_EXEC="server --server https://kubemaster1:6443 --token-file /tmp/token --tls-san $(hostname) --bind-address=${current_ip} --advertise-address=${current_ip} --node-ip=${current_ip} --no-deploy=traefik"
    export INSTALL_K3S_VERSION="v1.16.15+k3s1"
    sh /home/packer/k3s
fi

# Wait for node to be ready and disable deployments on it
sleep 15
kubectl taint --overwrite node $(hostname) node-role.kubernetes.io/master=true:NoSchedule

As you can see, no more key deployment, no more K3S download. I just configure my host and launch directly K3S which is already in my box.

The command line arguments (INSTALL_K3S_EXEC) are exactly the same.

Installation of worker nodes

#!/bin/sh
# Add current node in  /etc/hosts
echo "127.0.1.1 $(hostname)" >> /etc/hosts

# Add kubemaster1 in  /etc/hosts
echo "10.0.0.11  kubemaster1" >> /etc/hosts

# Get current IP adress to launch k3S
current_ip=$(/sbin/ip -o -4 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1)

# Launch k3s as agent
scp -o StrictHostKeyChecking=no root@kubemaster1:/var/lib/rancher/k3s/server/token /tmp/token
export INSTALL_K3S_EXEC="agent --server https://kubemaster1:6443 --token-file /tmp/token --node-ip=${current_ip}"
export INSTALL_K3S_VERSION="v1.16.15+k3s1"
sh /home/packer/k3s

We find here the same logic: no more download, basic configuration of the host and direct launch of K3S.

Traefik installation

The Traefik virtual machine is reduced to launching the binary directly by using the configurations pushed by Vagrant.

#!/bin/bash
# Run Traefik as a front load balancer
/home/packer/traefik --configFile=/tmp/traefikconf/static_conf.toml &> /dev/null&

We run the whole thing

So we have everything we need to launch the deployment with Vagrant with a simple

vagrant up

As for my previous post, after a few minutes my cluster is online. It is possible to check that everything works well with the following commands:

source ./scripts/configure_kubectl.sh
kubectl get nodes 

You should get a return similar to this one:

To conclude

This article stays on a very basic use of Packer. The tool is very powerful and allows you to have multiprovisioning approaches in a very simple way. Moreover, the strength of Packer, like most of the infra as code tools, is its reproducibility: it is very easy to recreate identical base images without having to share files of several hundred MB.

Like many of HashiCorp's tools, the language is quite high level and very accessible. As for me, it took me an hour to make my first VM from scratch a few years ago using Packer.

Don't hesitate to react via the comments if you have any questions or remarks !

On December 10th, one of the most anticipated games of the year was released: Cyberpunk 2077.

Personally, I enjoy playing it, but it’s not the case for everyone.

With an outside look of an IT professional, I suggest you see today the “mistakes” that I think were made in this project, and how some mistakes could have been avoided. This article is not about the game itself, but rather about the organizational and technical aspects.

A little background

The studio

Cyberpunk is developed by CD Projeckt Red, a rather small Polish studio (on the scale of the titans of the present industry). It is a company of about 800 people (according to their website). It is the studio behind the critically acclaimed The Witcher franchise, most recently adapted into a series by Netflix.

It is also the GOG.com platform, bought a few years ago to sell games from independent studios. GOG stands out for its model promoting the fact of selling games without DRM (copy protection) which can sometimes cause problems.

It is a company that historically worked on a single license: The Witcher.

The game

Development of this game would have started in 2012, according to a press release at the time. The first teaser released in 2013 not specifying a release date.

Developed a lot in the background, little information is released over the years, we start to hear about the game again at the E3 of 2018 (the video game mecca). A first video offering an overview of the game is then presented.

Initially scheduled for April 16, 2020, the release will be postponed 3 times: first on September 17, 2020, then on October 27, 2020, to finally release on December 10, 2020.

The development of the game would have cost about 270 million euros, one of the most expensive in the video game industry, similar in size to GTA V, for example.

However, its release was quite catastrophic for the image of the studio: the game seems poorly finished, one feels that some elements have been “botched”. Worse, the game is unplayable on PS4 and Xbox One, which were the existing consoles at the beginning of the development. The game is punctuated by a lot of bugs more or less blocking.

Also, the game doesn’t really look like the trailers presented earlier, it seems much more rigid, the player having much less freedom than announced, especially on the character customization part.

This had several impacts:

• Removal of the game from online stores
• Massive refund campaign
• Huge impact on the company’s image
• Impact on the company’s stock market valuation

Agility, don’t know it!

I’m not going to present all the concepts of agility here, but rather talk about one of the basic objectives: to bring the business and the developers closer together so that each can take into account the constraints and needs of the other.

What is clear in the history of the development of this game is the total desynchronization of the two parties. The successive postponements are a typical example, which highlights the fact that the business is not aware of the remaining workload or that it has been greatly underestimated.

In the first case, it shows that the approach taken is not consistent to deliver a finished product, in the second case, it may come from a lack of guidance.

In an agile/DevOps world, measuring is key. Normally, we are able to roughly estimate the velocity we can achieve in a sprint, although mistakes can happen.

”We ignored the signals about the need for additional time to refine the game on the base last-gen consoles. It was the wrong approach and against our business philosophy.”

Yet, we can see in the official releases that alarms were raised about the developers needing more time to release a finished product.

Even if one can easily understand the “deadline” of December 10 (2 weeks before Christmas), it is a pity that this is such a strong deadline that one ignores all the signals that indicate that one is heading for the wall.

The importance of testing

Releasing a game on four consoles simultaneously and on computers, as well as on streaming platforms, is not as simple as it seems:

• Multiple technical specifications
• Specificity of the SDK of each console
• Different processor architectures

CD Projekt Red admitted that they did not test their game enough on consoles and focused on the PC.

The concern is that even on a PC, some bugs and anomalies are very complicated to understand, we could mention:

• The non-support of SMT procedures on AMD processors: just incomprehensible that the tests are conclusive, to have tested on the config we have at home (AMD Ryzen 1600 + Nvidia GTX 1070), we had a 40% gain of additional performance (about 20 FPS)! (This point has been corrected since version 1.05)
• The default QWERTY configuration and the non-ergonomic controls in general
• The elements of the scenery that float in the air (phones for example)…
• The “Lorem ipsum” that hangs in some languages, these placeholders are normally used during development to simulate a text fill

• Bugs with subtitles that remain permanently displayed on the screen or object tooltips
• Vehicles that appear under the map

And I could continue the list for a long time… Knowing that these are only the bugs that I could see on my side with a recent PC!

And the security of all this?

Cyberpunk is launched via a launcher, which fortunately can be bypassed when you have it, like me, on Steam. The problem is that this launcher is (like many launchers) a huge security flaw, allowing to easily execute arbitrary code if compromised…

As far as hacking is concerned, we’re in trouble “script kiddies” here…

But since we don’t have a DevOps approach, I won’t ask for DevSecOps!

What could have limited this disaster

I know it’s easy to criticize after the fact; however, I’d like to bring a few points that might have helped limit this disaster.

Limiting the release to PC at first

This is the strategy used by Rockstar for its games, release first on console, then a few months later on PC.

This allows having several gains:

• Bugs are located on fewer platforms
• It allows focusing its development and testing on fewer platforms
• It allows to potentially sell several copies of the same game since players are not necessarily ready to wait several months

CD Projekt Red is a small studio, to spread out on so many platforms in parallel is risky.

Not selling the perfect game in all aspects

The game has clearly been oversold, announced several times as “fully playable, but needs to be refined”, we find the aspect of desynchronization between business and developers, where we first sell things and then look at how to develop them.

This is something I’ve often experienced in IT (unfortunately), where you promise things to a customer, but at no time:

• IT was asked for a realistic estimate of the timeframe
• We asked ourselves if it was technically feasible without costing more than it brings to the company.

The problem is that the game was sold on too many aspects at the same time, which I think are difficult to do for a company of this size.

Release the game in “early access” or “closed beta”

Method used more and more often, it would have been possible to release the game in “early access”, that is to say, by releasing it “as is”, but by iterating quickly. Players are more understanding of games that have problems in early access than a final version, and this allows for much more feedback.

Also, a closed beta with a limited number of users would probably allow solving a lot of problems (especially if you take different game configurations).

Not everything is black

To finish this post, I would conclude by saying that everything is not black.

On PC, the game is completely playable, I have for my part quite a few hours of play on it. Graphically, it is also a success, with a universe of its own, the art direction has done an impressive job.

Also, in terms of writing, we feel that some characters are quite worked and have depth. I also salute the efforts of the studio to put many women as central characters in their game!

The Witcher was also riddled with bugs at its release, but is now considered one of the best games of the decade, no doubt that the shot will be corrected on this game, especially since the studio has already delivered its first salvos of fixes.

However, I would also note that releasing “poorly finished” games under the pretext that they can be patched later is becoming a norm in the video game industry, making the customer suffer for having an unstable product. Nowadays, many studios are in the same situation (hi Assassin’s Creed and its blocking bugs that were fixed one month after its release).

Computer attack patterns have evolved in recent years. Cryptolockers have become the spearhead of many hackers.

Does your antivirus vendor promise you that you are protected against these new threats? OK, prove it before you get stuck by a real attack.

Let’s talk about cryptolocker

The principle of a cryptolocker is quite simple: encrypt target files (often.doc, .txt, . odt, etc.) and then demand a ransom. A ransomware has nothing to gain by destroying the underlying OS, so system files are rarely touched.

So it will encrypt as many files as possible and then ask for payment (often in Bitcoin or other cryptocurrencies) to obtain the decryption key.

These threats are very serious, large groups have had to face them recently and were sometimes paralyzed for several days.

Detecting a cryptolocker

The concern is that we are nowadays in a world with more and more sophisticated and discreet malware. Nowadays, the power of heuristic engines on antivirus software is widely exploited. These engines detect abnormal or dangerous behavior rather than just a previously identified binary (signature).

The purpose of the test I’m offering today is to test the performance of this engine, whether it’s on its detection speed, reaction (blocking the attack) and its remediation.

Ransomware as a service
To perform these tests, we will use a small open-source tool:

GitHub - leonv024/RAASNet: Open-Source Ransomware As A Service for Linux, MacOS and Windows

Open-Source Ransomware As A Service for Linux, MacOS and Windows - GitHub - leonv024/RAASNet: Open-Source Ransomware As A Service for Linux, MacOS and Windows

GitHubleonv024

This small python script simulates the basic operation of a cryptolocker.

It will allow you to start a local server, which will be used to collect information about the encryption (IP address, encryption keys, etc.).

It also allows you to create the script that will be used for the encryption, and to compile it to make it portable.

Why is this project interesting?

First of all, the interest is already that it is completely harmless:

• It does not spread on your network
• You can decrypt the data
• It is open source, you can see or modify the code
• it works on all OS, because it only requires Python 3

Then, the attack model is all the more interesting as it is realistic, the use of a python script in a “standalone” mode allows simulating a concrete case: a third party library available on pip for example, which would be malicious.

This also allows you to check that your antivirus software reacts to attacks that are not binaries.

Download and set up the prerequisites

Clone the repository

If you have the Git client installed, you can directly clone the repository via the Git client:

git clone git@github.com:leonv024/RAASNet.git

If you don’t have the Git client, you can also directly download the zip file from the repository.

Installing the prerequisites

On the PC on which you want to test the script, it is necessary to have Python 3 installed, as well as the pip3 client.

For these installations, I invite you to go to the associated documentation pages to see how to install them according to your OS context:

• Python 3: https://docs.python.org/fr/3/using/index.html
• PIP 3: https://pip.pypa.io/en/stable/installing/

Once these steps are made, it is necessary to install the prerequisites:

pip install -r ./requirements.txt

Under Windows, it is necessary to install an additional package for the management of windows:

pip install image

Start the application

Once the prerequisites are installed, you can start the application:

python3 ./RAASNet.py

Once this is done, you will have to create an account, it is not necessary to have a valid email address for this account.

Then you can start the server, this will allow you to collect information when encryption starts.

Warning: If you do not start the server, you will not have the decryption key!

Initiate the attack payload

Now that we have prepared the base of our attack. The next step is to create the attack script.

To do this, we will click on “Generate Payload”. Then you can select the options for your attack:

• The list of target files
• The base directory for the attack
• The encryption method to use
• Whether or not to display a window at the end to indicate that the content is encrypted

Once the configuration is done, just click on “Generate” to create two scripts:

• payload.py: this script is your attack load itself
• decrypt.py: this script allows decrypting the files after the attack

It’s popcorn time!

Now that you have everything ready, it’s the most fun part: launching the attack!

Here are three points to watch on your antivirus:

• whether or not the attack is detected
• the speed of detection
• the possibility to remedy or not: does the antivirus know how to restore the encrypted files?

To launch the attack, a simple command:

python3 payload.py

Depending on the encryption method, you have chosen and the volume of data, the encryption will take from a few seconds to several minutes.

The interest here is to see if your antivirus software reacts or not to the encryption, via its heuristic engine.

For the test to be as consistent as possible, don’t hesitate to vary the attack mode, for example by launching the encryption from a USB key and remotely setting up the server.

Moreover, to be sure to trigger your antivirus, don’t hesitate to put a little bit of file volume (a few thousand) to make sure that the volume is sufficient to make it react.

And in binary?

RAASNet also allows you to convert python scripts into classic binaries. Including replacing the basic icon.

By doing so, you can reproduce classic attack patterns, e.g., a PDF invoice by email, an autorun executable on a USB stick, a download from a third-party site, etc.

Once again, the idea is to reproduce classic attacks and observe the reaction (or not) of your antivirus.

To conclude

Whether your antivirus has detected the attack or not, you must remain pragmatic.

Depending on the vector used, it may be normal that your antivirus considered the action as legitimate, that’s why you should not hesitate to transfer the viral load on a simple USB key, for example.

Moreover, this malware is voluntarily unknown and harmless, so the risk is controlled.

However, this is a simple and interesting test to do to check the reaction of your antivirus.

For my part, my personal antivirus detected the abnormal behavior after a few encrypted files and restored the content.

A few days ago, an incident impacting the AWS cloud provider had a significant impact on many companies and services directly affected by this instability.

I saw on social networks many reactions, often beside the subject (unfortunately) and I thought it could be useful to give you my analysis of the subject.

Rewind

Let’s start by recalling the incident a bit.

On Wednesday evening (French time), AWS encountered a growing number of errors in some of its services in the us-east-1 (North Virginia) region.

Although it is considered that an incident in a single area is not supposed to have a huge impact on AWS, it should be considered that this area is the “core” area of AWS. Most global services (IAM, CloudFront, Route53, etc.) depend heavily on it.

It should also be taken into account that it is considered today that about 30% of the Internet depends directly on AWS, which is just colossal.

This is not the first AWS incident, and it won’t be the last. It’s been in the news because of the length of time it’s been going on and the visibility it’s been given. A lot of companies have indeed cleared themselves of any worries by sending the ball back to Amazon, but things are a little more complicated than that.

“Everything fails all the time.”

This sentence does not come from me, but from the CTO of Amazon Web Services, Werner Vogels.

It is based on an elementary rule: wherever you host your service, you have to take into account that it will sometimes fall down, like any infrastructure.

Having been in production for more than 10 years, I’m familiar with the subject: you can redund a service, you just reduce the risk of an incident. There is no such thing as zero risk.

During my AWS training sessions, this is a point that often comes up, the question to ask yourself about the design of your infrastructure is not “how to prevent my infrastructure from falling down”, but rather “what to do when my infrastructure falls down”.

Those of you who, like me, work on AWS on a daily basis know that there are incidents every day, less impacting, but there are still incidents.

The risks of hyperconvergence

The public clouds and especially the GAFAMs have allowed a hyper-convergence of many services.

Indeed, because of the bricks proposed, it is very easy to say to oneself that one is going to place all one’s logs at AWS/Azure/GCP.

A company can indeed find something to meet almost all its needs of the shelf, very often with directly managed services and the support that goes with it.

This aspect has been an important vector of acceleration in recent years, allowing us to deploy and therefore innovate ever faster.

The concern is that this has also made AWS an Internet colossus. Many companies have become completely dependent on this provider.

But to just say that the concern is there is to see only the tip of the iceberg.

Design and risk assessment

In an IT department, the role of an architect is to design a resilient infrastructure that meets technical and functional needs.

When we talk about resilience, we often think of always having an application running 24/7 with an availability rate of 99.99%.

In reality it’s more complicated than that.

Risk assessment VS. cost

Normally, when designing an architecture, one asks oneself the question of the targeted availability rate as well as the cost of unavailability.

In addition, we need to see if there are mechanisms to mitigate unavailability.

For example, let’s imagine a home automation device: it sends its metrics to AWS every 10 minutes.

What happens if my service that has to receive these metrics is unavailable?

2 choices here:

• I lose my metrics: the availability of my service is therefore essential
• I have a buffer locally on the device, allowing delaying in case of unavailability: in this case, I can afford unavailability, because I potentially lose only the “real time” part

These two choices are already determined by the amount of money you want to put into a project.

Having local storage costs additional hardware, additional assembly complexity, and additional development and maintenance costs.

Also, it means that my server is able to manage in “catch-up” mode.

The other aspect is to ask what the cost of downtime is:

• In terms of service provided
• In terms of potential contractual penalties
• In terms of image

We then put this unavailability against the probability of the latter. Very often, simplistically, the potential availability of infrastructure is that of the element with the lowest availability.

Then, we can ask ourselves how much it will cost to cover this unavailability.

Then we make a simple comparison, and quite obviously, the most cost-effective solution is the one that is taken into account.

This is why unavailability is not always a problem. It is the role of the architects to evaluate it upstream.

The chaos theory

To have confidence in your infrastructure, you have to test it. This is the basic idea of chaos engineering.

Starting from the observation that your infrastructure is going to fall is a point, to make it fall voluntarily, in a framework that you can control:

Increase confidence in your high availability
to test your automated procedures and/or remediation.
Tools exist for this purpose, if you want to know more, I invite you to read the excellent posts of my colleague Akram on this subject on the WeScale blog [French links]:

Le guide de Chaos Engineering : Partie 1

Le Chaos Engineering (CE) est une discipline d’expérimentation dans les systèmescomplexes et distribués, qui vise à se rassurer sur leur comportement face à desincidents et perturbations. Dans cet article je vais m’attacher à simplementapporter les bases du chaos et quelques définitions pour pose…

Le blog des experts WeScale - DevOps, Cloud et passionAkram RIAHI

Le guide de Chaos Engineering : Partie 2

Introduction Nous avons vu dans la première partie du guide du Chaos Engineering (CE)[/2019/09/26/le-guide-de-chaos-engineering-part-1/] que le CE ne peut pas avoirlieu sans expérimentations. Nous allons continuer notre voyage fantastique dans ce monde en passant par lepays du “Chaos Tools Cou…

Le blog des experts WeScale - DevOps, Cloud et passionAkram RIAHI

Some companies go so far as to use these tools in production. For example, Netflix uses its simian army in production to ensure the resilience of its infrastructure.

Destroying while mastering also allows you to better know how to react when your infrastructure falls, as it is a habit.

To conclude

The companies that put the blame on AWS are the only ones at fault.

Either they underestimated the impact of unavailability, or the unavailability costs less than expected.

Either way, it is a design choice to depend on a single supplier and a single availability area. Blaming one supplier is cowardly and does not represent the reality of things.

It is possible to prepare and simulate these unavailabilities in advance of the phase in order to react in the best possible way; repetition creates trust.

The other concern also comes from the fact that many companies have bet everything on AWS, the incidents of the last few days may be going to reshuffle the cards a bit, but is it worse?

Advocating for a sovereign cloud provider is not the answer either if you put all your marbles in one place. Once again, good infrastructure design is essential, better safe than sorry. You can learn more about this subject with Damy. R’s post [French link]:

AWS tombe, l’internet tremble · Damy.R

Damy.R Réparateur de nuage passionné par l'opensource et la mouvance DevOps.

Testing Kubernetes is quite easy thanks to solutions such as Minikube.

However, when you want to test cluster-specific features, such as load balancing or failover, it is not necessarily suitable anymore.

It is possible to build your Kubernetes infrastructure on servers, or by using managed services from a cloud provider (Kapsule at Scaleway, AKS at Azure, GKE at GCP or EKS at AWS for example).

Nevertheless, these solutions cost money. When you just want to test functionalities or self-training, it's not necessarily appropriate.

In this post, I propose you see how you can set up a Kubernetes cluster locally on your computer with a similar behavior as a classical cluster.

To do this, we will use several tools that I will describe below:

• Vagrant
• VirtualBox (or VMWare)
• K3S
• Traefik

Vagrant: Provisioning of virtual machines

Vagrant is a tool from HashiCorp (editor of Terraform I already mentioned).

This program allows to quickly deploy virtual machines by exploiting description files.

Thus by writing a VagrantFile, it is possible to deploy in a few minutes one or more machines, by provisioning them with scripts or tools such as Ansible.

The advantage of Vagrant is that it allows configuration sharing to allow a complete team to work under the same conditions locally, reproducing a low-cost production behavior.

A simple text file is enough to share. Moreover, it is in fact possible to version this file on a version management tool.

K3S: The lighter kubernetes

K3S is a tool created by Rancher (who also created an orchestrator of the same name for Docker).

It is a lightweight Kubernetes that can work on smaller configurations. It can work too without any problem on a raspberry pi.

In our case, it will allow us to override the limitations of K8S which requires at least 2G of ram to run.

Since we are going to create several servers, the idea is to limit the memory needed to the maximum.

In this post we will exploit the multimaster mode, which is very recent. If you want more information on this subject, I invite you to read the excellent article by my colleagues at WeScale [French Link].

Traefik : Again and again the load balancer

In order to reproduce as closely as possible the behavior of a classic cluster, we will also deploy a Traefik server as a front end, in front of Kubernetes, which will balance the load between the 3 master nodes.

The choice of Traefik is arbitrary, any reverse proxy will do.

Once again, the goal is to have something light, hence the use of Traefik which is very resource efficient.

Target deployment overview

We are going to have 3 distinct layers in our deployment :

front_lb : A machine using Traefik to manage the incoming traffic to Kubernetes
kubemaster# : The 3 kubernetes "master" servers
kubenode#: The 3 kubernetes servers serving as execution nodes for the pods

Prerequisites and information

The deployment I will describe below has been tested and validated with the following configuration:

The deployment does not normally use any of this information on a hard disk, but some settings may need to be adjusted depending on your host system.

Also, in order to allow my nodes to communicate with each other in SSH, I created a RSA key that I deploy automatically.

This key is in the ./.ssh directory, and it is necessary that you create it yourself before starting the deployment via Vagrant.

You can do it easily with the following command:

ssh-keygen -f ./.ssh/id_rsa

At the passphrase request, it is necessary not to put any, because we will use this key to have a scripted connection between our nodes.

It's time to get our hands dirty!

All right, enough talk, when do we deploy?

Not right away, first of all, let's look at what we're going to deploy.

The basic OS

In our machines, we will first have to deploy an operating system.

Vagrant uses a system of "boxes" which are actually images prepared for Vagrant. It is possible to get the list of boxes on the official website.

In our case, we will use a Ubuntu 18.04 box.

I'm not a fan of Ubuntu as a server, I prefer a good old Debian or Centos, but Ubuntu is the system officially supported by K3S.

For Traefik, I'll use the same box, just because I'm lazy. In itself, nothing prevents me from using another base box.

Masters

So we have at first the Kubernetes "masters" servers.

For the latter, we have two distinct cases.

The first server must use a parameter to indicate that we want to initiate a K3S cluster, with the "--cluster-init" parameter.

Then, the other nodes will have to connect to the first node with the secret generated by it.

To simplify the exchange of the secret, the second and third nodes will then download the file containing the secret via SCP (hence the creation of the SSH key).

In addition, we will tell K3S the IP address of the servers, because the VM has several network cards, so K3S would mount the wrong IP each time.

So here is the associated Vagrant configuration block :

MASTER_COUNT = 3
IMAGE = "ubuntu/bionic64"

...

Vagrant.configure("2") do |config|

  (1..MASTER_COUNT).each do |i|
    config.vm.define "kubemaster#{i}" do |kubemasters|
      kubemasters.vm.box = IMAGE
      kubemasters.vm.hostname = "kubemaster#{i}"
      kubemasters.vm.network  :private_network, ip: "10.0.0.#{i+10}"
      kubemasters.vm.provision "file", source: "./.ssh/id_rsa.pub", destination: "/tmp/id_rsa.pub"
      kubemasters.vm.provision "file", source: "./.ssh/id_rsa", destination: "/tmp/id_rsa"
      kubemasters.vm.provision "shell", privileged: true,  path: "scripts/master_install.sh"
    end
  end

...

end

• Line 8: Vagrant loop structure
• Line 11: We name each machine with a different host name.
• Line 12: Each machine will have a fixed IP allocated (from 10.0.0.11 to 10.0.0.13)
• Line 13/14: We push our SSH key which will be deployed by the script indicated in line 15

As you can see, the description of the Vagrant side is quite basic.

Now, let's see the second side of this deployment: the provisioning shell script.

#!/bin/sh

# Deploy keys to allow all nodes to connect each others as root
mv /tmp/id_rsa*  /root/.ssh/

chmod 400 /root/.ssh/id_rsa*
chown root:root  /root/.ssh/id_rsa*

cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
chmod 400 /root/.ssh/authorized_keys
chown root:root /root/.ssh/authorized_keys

# Add current node in  /etc/hosts
echo "127.0.1.1 $(hostname)" >> /etc/hosts

# Get current IP adress to launch k3S
current_ip=$(/sbin/ip -o -4 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1)

# If we are on first node, launch k3s with cluster-init, else we join the existing cluster
if [ $(hostname) = "kubemaster1" ]
then
    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --cluster-init --tls-san $(hostname) --bind-address=${current_ip} --advertise-address=${current_ip} --node-ip=${current_ip} --no-deploy=traefik" sh -
else
    echo "10.0.0.11  kubemaster1" >> /etc/hosts
    scp -o StrictHostKeyChecking=no root@kubemaster1:/var/lib/rancher/k3s/server/token /tmp/token
    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --server https://kubemaster1:6443 --token-file /tmp/token --tls-san $(hostname) --bind-address=${current_ip} --advertise-address=${current_ip} --node-ip=${current_ip} --no-deploy=traefik" sh -
fi

# Wait for node to be ready and disable deployments on it
sleep 15
kubectl taint --overwrite node $(hostname) node-role.kubernetes.io/master=true:NoSchedule

Once again, let's cut out this script :

• Lines 4 to 11: We deploy our SSH key and add it as an authorized host.
• Line 19 to 27: If we are on the node "kubemaster1", we launch a cluster-init, otherwise we join the existing cluster.
• Lines 30 and 31: we wait 15 seconds for the node to be functional, then we "taint" it to indicate that the node should not execute pods. In fact, by default, since K3S is designed to work on small systems, the master runs in standalone mode and can also execute pods.

Note also that I disable the installation of Traefik, indeed, our nodes being masters, they are not supposed to run an Ingress Controller.

Execution nodes

In the same way, let's look at the contents of the VagrantFile as far as our execution nodes are concerned.

MASTER_COUNT = 3
IMAGE = "ubuntu/bionic64"

...

Vagrant.configure("2") do |config|

...

  (1..NODE_COUNT).each do |i|
    config.vm.define "kubenode#{i}" do |kubenodes|
      kubenodes.vm.box = IMAGE
      kubenodes.vm.hostname = "kubenode#{i}"
      kubenodes.vm.network  :private_network, ip: "10.0.0.#{i+20}"
      kubenodes.vm.provision "file", source: "./.ssh/id_rsa.pub", destination: "/tmp/id_rsa.pub"
      kubenodes.vm.provision "file", source: "./.ssh/id_rsa", destination: "/tmp/id_rsa"
      kubenodes.vm.provision "shell", privileged: true,  path: "scripts/node_install.sh"
    end
  end

...

end

As you can see, the content is pretty much the same as for the masters.

So the only differences are :

• The IP pool, we are this time between 10.0.0.21 and 10.0.0.23
• The script executed for provisioning

Let's take a look at provisioning:

#!/bin/sh

# Deploy keys to allow all nodes to connect each others as root
mv /tmp/id_rsa*  /root/.ssh/

chmod 400 /root/.ssh/id_rsa*
chown root:root  /root/.ssh/id_rsa*

cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
chmod 400 /root/.ssh/authorized_keys
chown root:root /root/.ssh/authorized_keys

# Add current node in  /etc/hosts
echo "127.0.1.1 $(hostname)" >> /etc/hosts

# Add kubemaster1 in  /etc/hosts
echo "10.0.0.11  kubemaster1" >> /etc/hosts

# Get current IP adress to launch k3S
current_ip=$(/sbin/ip -o -4 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1)

# Launch k3s as agent
scp -o StrictHostKeyChecking=no root@kubemaster1:/var/lib/rancher/k3s/server/token /tmp/token
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --server https://kubemaster1:6443 --token-file /tmp/token --node-ip=${current_ip}" sh -

This time we have a much simpler script!

The first part always consists of deploying our SSH keys.

So the last line launches K3S in "agent" mode by telling it to connect to the kubemaster1 node (I chose this node arbitrarily, any of the 3 masters would have done).

Load balancer frontend: Traefik

Finally, we deploy our front load balancer.

This installation is very basic, since we have no production constraints, the need is not to be as secure as a productive environment, but simply to have a load balancer to reproduce the normal behavior of a Kubernetes cluster.

On the Vagrant side, we are still very basic:

MASTER_COUNT = 3
IMAGE = "ubuntu/bionic64"

...

Vagrant.configure("2") do |config|

...

  config.vm.define "front_lb" do |traefik|
      traefik.vm.box = IMAGE
      traefik.vm.hostname = "traefik"
      traefik.vm.network  :private_network, ip: "10.0.0.30"   
      traefik.vm.provision "file", source: "./scripts/traefik/dynamic_conf.toml", destination: "/tmp/traefikconf/dynamic_conf.toml"
      traefik.vm.provision "file", source: "./scripts/traefik/static_conf.toml", destination: "/tmp/traefikconf/static_conf.toml"
      traefik.vm.provision "shell", privileged: true,  path: "scripts/lb_install.sh"
      traefik.vm.network "forwarded_port", guest: 6443, host: 6443
  end
end

This time, no SSH key, we rather push the Traefik configuration that I will describe below.

Then, we run the installation. In addition, we indicate to map port 6443 of the host (my machine) to 6443 of the virtual machine. This will allow me to run kubectl commands from my host through Traefik.

For Traefik, we have two configuration files :

• The static configuration: This is the basic configuration of Traefik, this is where we will indicate the listening port for example
• The dynamic configuration: This is where you will find the information that Traefik can collect on the fly, including the configuration of the endpoints.

Let's see our configuration.

Static configuration :

[entryPoints]
  [entryPoints.websecure]
    address = ":6443"
    [entryPoints.websecure.http.tls]
      [[entryPoints.websecure.http.tls.domains]]
        main = "10.0.0.30"

[providers.file]
  directory = "/tmp/traefikconf/"

[serversTransport]
  insecureSkipVerify = true

So we can see :

• That I listen in https on 6443 (default port of Kubernetes)
• I then indicate in which directory my dynamic configuration is located.
• I indicate not to check the certificate; indeed, my K3S server uses a self-signed certificate by default.

Then, the dynamic configuration, this is where I will define my endpoints. Note that any modification of this one is taken into account by Traefik instantly.

[http]

  [http.routers]
    [http.routers.routerTest]
      service = "k3s"
      rule = "Host(`10.0.0.30`)"

  [http.services]
    [http.services.k3s]
      [http.services.k3s.loadBalancer]
        [[http.services.k3s.loadBalancer.servers]]
          url = "https://10.0.0.11:6443"
        [[http.services.k3s.loadBalancer.servers]]
          url = "https://10.0.0.12:6443"
        [[http.services.k3s.loadBalancer.servers]]
          url = "https://10.0.0.13:6443"

Nothing original here either. First, I define a Traefik k3s service, to which I say to send any request that arrives with the host "10.0.0.30" (the IP address I set to Vagrant).

Then I define a load balancing on the 3 Kubernetes masters. Traefik's default behavior is the "round robin" mode, which means that it will send requests on each node one after the other, regardless of a possible load.

Finally, let's have a look at the installation shell script :

#!/bin/sh
# Download and deploy Traefik as a front load balancer
curl https://github.com/containous/traefik/releases/download/v2.2.11/traefik_v2.2.11_linux_amd64.tar.gz -o /tmp/traefik.tar.gz -L
cd /tmp/
tar xvfz ./traefik.tar.gz
nohup ./traefik --configFile=/tmp/traefikconf/static_conf.toml &> /dev/null&

This is the simplest of the three scripts. As you can see, I directly download the Traefik binary (version 2.2.11), because Traefik is also a standalone binary written in GO.

Then, I launch Traefik by detaching it from the terminal and simply telling it where its static configuration file is.

It's time to launch all this!

You can find all the code used in GitHub :

teddy-ferdinand/vagrant-k3s-cluster

Déploiement d’une infrastructure K3S avec Vagrant. Contribute to teddy-ferdinand/vagrant-k3s-cluster development by creating an account on GitHub.

GitHubteddy-ferdinand

Install Vagrant

First we need to install Vagrant.

Here we have several solutions available:

Installation via the package manager: Often the simplest solution under Linux, not all distributions have Vagrant and it should be noted that the version can sometimes be (very) late on the official site. For my part, this is the installation I did.
Standalone binary : Vagrant is a standalone binary written in GO, it is also available on the official website, you can also install it this way.
These two installations won't change anything afterwards, so choose the one that suits you best.

Launch deployment

Once Vagrant is installed, go to the root directory, where our VagrantFile is located. So we should have a structure similar to this one:

We can now launch the deployment with a simple

vagrant up

You should normally see the deployment begin, with the download of the Ubuntu box.

From there, full deployment will take about 10 minutes.

Connect to our cluster

Once the deployment is complete, Vagrant normally gives you back your hand.

All we have to do is configure the Kubernetes client on our workstation.

I left a little script that will allow you to download kubectl, and configure it to connect to our cluster, without touching any configuration already present on the host.

Here is its content :

#!/bin/sh

# Get kubectl
curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /tmp/kubectlvagrant
chmod +x /tmp/kubectlvagrant

# Get password from master config file
PASSWORD=$(vagrant ssh kubemaster1 -c "sudo grep password /etc/rancher/k3s/k3s.yaml" | awk -F':' '{print $2}' | sed 's/ //g')

#Create kubectl config
cat << EOF > /tmp/kubectlvagrantconfig.yml
apiVersion: v1
clusters:
- cluster:
    server: https://10.0.0.30:6443
    insecure-skip-tls-verify: true
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    password: ${PASSWORD}
    username: admin
EOF

# Create temp vars to use kubectl with vagrant
export KUBECONFIG=/tmp/kubectlvagrantconfig.yml
alias kubectl="/tmp/kubectlvagrant"

As you can see, this script will :

• Download the kubectl client and place it in /tmp/kubectlvagrant making it executable.
• Then, we will retrieve the admin login password of Kubernetes.
• We then insert this password in a kubeconfig configuration template. We will note the "insecure-skip-tls-verify" parameter which indicates to ignore the certificate, because once again we are on a certificate self-signed by Traefik. The back end is my virtual machine with Traefik.
• Finally, I create an environment variable to tell kubectl where to get its configuration and an alias for kubectl to point to the one we downloaded.

To use this script, you just have to source it

source ./scripts/configure_kubectl.sh

Be careful not to execute the script, aliases and export would not work, you have to source it well.

To go back to the initial state, just reopen a new terminal.

It's time to test our cluster!

Once our script is executed, let's try to see our nodes.

kubectl get nodes -o wide

It is possible that this request may take a little longer than normal since the cluster receives its first connection.

This is good news! We have our 3 masters and 3 execution nodes!

We're going to do a little basic test, namely to deploy a 3 Nginx pod.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/content/fr/examples/controllers/nginx-deployment.yaml

Checking the deployed pods, we see 3 pods: 1 on each of our Kubernetes runtime nodes.

kubectl get pods -o wide

After about 30 seconds, they will go into "running".

As you can see, you can now use Kubernetes in a similar way to a production server.

Once you don't need the cluster anymore, you can destroy it with a simple

vagrant destroy -f

After a minute, all your machines are now destroyed.

To conclude

As you can see, Vagrant makes it easy to create work environments, which makes application development or testing easier.

I also chose to use Traefik as a load balancer to show that there is also a Traefik binary. Indeed, we often talk about its Docker image, but it is above all a binary in GO.

Similarly, it is quite easy to create coherent infrastructures using Vagrant to reproduce a production environment.

The advantages are those I mentioned before:

• It's fast (10 minutes to create a complete cluster)
• It doesn't cost anything, since the host is your machine.
• It is easily transmitted, for example within a team.

If you want to know more about Vagrant, I recommend the Xavki YouTube playlist which explores the tool well [French Link]:

TUTORIALS VAGRANT - EXEMPLES

Quelques exemples utiles pour apprendre à utiliser vagrant

YouTube

If you want to learn more about K3S, you can find content on the WeScale blog [French Link]:

k3s - Le blog des experts WeScale - DevOps, Cloud et passion

Le blog des experts WeScale - DevOps, Cloud et passionRomain BOULANGER

As always, feel free to comment! What do you think of Vagrant + K3S to have a local Kubernetes cluster?

Hackers … we often see them in movies and TV shows. These experts are able to hijack NSA satellites with a string and a nail clipper (#MacGyver)! (Cover image from the movie Die hard 4)

I decided today to tell you about hacking in "real life". I consider myself to be a white hat (an ethical hacker) and I’m going to tell you about the common methods a hacker uses. I will focus here on the simplest part: websites.

Spoiler alert: you may be disappointed, many methods are very simple.

Hack, what for?

Before talking about method and vector of attack, the right question is why hack?

To answer this question, there are, from my point of view, 2 sides of the same coin to consider, both types of hackers.

The white hat will try to find a loophole to signal it. Yes, but why report it?

• By conviction, personally I have already found major security holes by altruism, I’m in favor of a safer web for everyone.
• Because he participates in bounty bugs and will therefore receive a reward for reporting flaw (subject to certain criteria, such as non-disclosure within x days)
• By challenge, I am also in this case, the challenge to break, to bypass the security. The pleasure of solving puzzles more or less complex…

The black hat, on the other hand, will try to take advantage of his loophole. There are several solutions for that:

• Extract information and resell it on the black market. For example, an enriched contact (email address, password, phone number, first and last name) can sell for a few cents. Taking into account that many sites have thousands or even millions of contacts in their database, the price rises quickly.
• Selling confidential information or trade secrets
• Sell credit card/PayPal information, and so on…
• Making a site unavailable. Some offer DDOS "as a service", allowing for example to temporarily shut down a competitor to discredit it.
• Requesting ransom after encrypting all the data of a company or individual
• Turn the machine into a zombie, used in a botnet or for phishing emails.

The purpose of this article is not to be exhaustive, but to tell you the main reasons.

Observe to break better

We often talk about incredible hacks, but we often forget to talk about more basic hacks. I had the opportunity to get past the security of some websites simply by being an observer. For example, recently I managed to access very sensitive data on a website simply because of a display problem.

Indeed, one of the images had loaded badly, and I had the reflex of the good old CTRL+U (Display Source)!

What was not my surprise when I saw PHP errors, whose HTML rendering had been hidden… Then by going back to those errors, I discovered a debug module enabled on this production environment.

Via this debug module, I was able to trace the site configuration files, in which there were database connection strings, FTP server connection information and many other information, including confidential information.

This is just an example, but be aware that there are many such cases. Observing the source code or the behavior of a site gives clues on how to tamper with its operation. It is also possible to observe the behavior of the application using the development tools (F12). This makes it possible to observe XHR-type calls, which often correspond to API calls, when the latter are not sufficiently protected. It is sometimes possible to have them display data from other clients or information that is supposed to be protected.

In one of my previous missions, I had put my finger on a security anomaly of this type, by modifying an API call, I was able to access all the information of all the clients registered in the database (we were talking about several million clients in this case). Fortunately, I had identified this bug on a pre-production environment, and the bug was fixed before going into production.

A lot of hacking is in fact done by opportunism, simply because a flaw was visible. Surface attack remains the most common method!

Knowledge is power

Observing is good, knowing is just as useful. To know is to know several things:

• Knowing the target: being able to identify the CMS used, for example, or the type of server that runs the site.
• Know the common flaws
• Some common attack vectors: XHR, SQL Injection, etc.

Why is this important? As a demonstration, if I want to try to brute force a site, knowing if I'm on a WordPress or Drupal type CMS changes a lot, I'm not going to attack both in the same way.

Moreover, I'm going to try to look for the latest security patches to attack what they fix. In production, it is common for servers not to be updated as soon as a patch is released (except for big security patches announced by the editor beforehand).

In the same way, knowing that a server is running on an obsolete version of Apache HTTPD (for example once again) allows me to know how I can compromise it to potentially take control of it.

In the world of hacking, technology watch is essential. You need to keep up to date with the latest major vulnerabilities, the latest methods, and the most common techniques.

For example, Amazon recently suffered an attack of several Tb per second. Everyone has been focusing on volume, yet on the scale of AWS, it's a grain of sand. What is interesting is to understand the attack vector: the UDP reflecting. Indeed, knowing how someone generated such a bandwidth is much more useful than a "clickbait" volume.

The right tool for the right attack

Of course I will not expose here all the tools to make an attack, that's not the point, I will just talk about common tools that can be used quite easily.

Detection of CMS/solutions used

I'd already be talking about the Wappalyzer extension. This browser extension allows you to have a lot of information at a glance, simply based on the headers and the presence of certain key files. Here is an example of this site:

Preparing your attack

Here I see the undisputed master, I have named the one, the only, the great MetaSploit. This program, which can be used for free, allows exploiting payloads or to create one's own to exploit known flaws. An article about how to use the tool will soon arrive on this blog!

Be the least visible on the Internet

I think everybody expects it, but for me, I can hardly see being more effective than Tor at this point. Why Tor instead of VPN? Most VPN providers log your activity, and even if they sell you the fact that they don't, you can never be 100% sure.

As far as tor is concerned, there are some reliable alternatives for me:

• Tor browser: the official Tor browser based on Firefox.
• Brave: The privacy-oriented browser, which has a Tor mode (a solution I personally use)

Note that tor is automatically blocked on some sites, just like VPNs.

Analyzing network behavior

Analyzing the network behavior can also be a good approach, for that, the most adapted tool is, in my opinion, the TCPDump/WireShark combo. On Windows, you can use netsh or Microsoft powertools.

The interest in this kind of case can be to see with which server we communicate, which information we send and receive and according to which protocol.

Have a "toolbox" OS.

For the lazy (as I am), it is also possible to directly exploit an OS. I see two candidates here:

Kali Linux: The Debian-based OS is recognized in the cyber security community, the "by offensive security" model has nothing to prove. It has the advantage of having a big community of enthusiasts who feed it and put a lot of tutorials around it. For my part, it's my main Linux.

Parrot Linux: Distribution also based on Debian. This distribution has the same advantages as Kali, a ready-made packaging, a strong community. I just find that the community around the product is a bit less present (but that's just my point of view, I may be wrong).

Note that the two Linux mentioned above are about the same age. They were both released in 2013.

Attackers everywhere, what to do?

I've already said it several times during my missions, when I'm asked how to avoid being attacked, the answer is simple: it's impossible!

The Internet is a jungle, there's not much you can do about it.

The best solution remains surveillance and reactivity. You have to keep in mind that black hats will often remain discreet until they can launch their attacks, they can wait months before showing up. The goal in this case is to accumulate as much information as possible, or to infiltrate as far as possible into the information system.

Similarly, many attackers will not launch an attack at 3 Gb/second, which would only be of interest to be very visible.

So what do we do? Personally, I see three axes:

• Metrology and supervision: finely supervising a production infrastructure is essential, it allows us to detect abnormal behavior, the more information and hindsight we have, the easier it is to detect patterns that are out of the ordinary.
• Regularly auditing applications: at this point, I am not necessarily talking about using an external company to audit security, but rather about auditing the authorizations, the open flows of an application yourself. Is it still necessary for my application A to access application B? Does it need to have as much access, to see when the last uses were made, and so on?
• Test as much as possible: I already talked about it on this blog, DevSecOps must become the basis of your applications, testing security as soon as your application is developed is essential.

The most complicated will always remain to detect weak signals, i.e., discrete attacks. I have 2 examples in mind on this subject:

• A "brute force" attack on ADFS, via a botnet network, which made 5 attempts per minute, therefore lost in the "normal" flow of several hundreds or even thousands of connections over the same interval.
• A compromised machine that was infected by a cryptographer, who was mining Bitcoin using 10% of the CPU.

It is also possible to equip yourself accordingly. On AWS, the GuardDuty service does an excellent job of detecting this type of behavior.

As said earlier, a malicious attacker will always try to be as discreet as possible.

To pay the ransom or not?

Lately, a ransomware attack got a lot of attention. Garmin was indeed targeted by a cryptolocker and has been immobilized for over ten days. Looks like the company ultimately decided to pay the ransom.

$10,000,000 is the ransom price… Like the debit story, the amount is anecdotal for a company Garmin's size.

Still, from where I stand, paying ransom is always a bad idea:

We're talking about hackers, they have absolutely no obligation to unlock the data
If Garmin decided to pay today, what's to stop another company from doing so?
It is possible that another backdoor will be present in the system and lock again the company in a few months.
Paying encourages these attacks because it sends a signal that it's an effective way to make money.

To conclude this long article…

Rather than a conclusion as long as this article, I propose bullet points on the misconceptions about hacking:

• Hackers don't need 15 monitors and 8 computers…
• Hackers very rarely hack from mobile phones, at best it is used as an attack vector.
• Hacking a system is not done in a few seconds (except in very special cases), and sometimes takes several weeks or months.
• Social hacking is also a more and more common way to get in, as I have talked about it in the past.
• In movies and series, hacking is often used in Deus Ex Machina to avoid having to manage a too complicated story, computer science is often seen as an end and not a tool.
• Most of the movies and series that have hacking scenes often make me laugh (except Mr. Robot which is devilishly realistic).
• I don't dress in a hoodie to hack into systems
• Hackers are not all outcasts of the system and many go unnoticed, they just decided to play with their rules on the Internet.

If you want to train without risk (and without the risk of jail), I can only recommend the excellent hackthebox that will allow you to use your skills and sharpen your hacker talents. This site provides sandbox environments for hackers. Please note that you will need to "hack" the site in order to create an account.

I have been using AWS professionally for over 4 years now.

To be a bit old-fashioned, when I started on AWS, the following services and features did not exist:

• The ALB/NLB
• ACM
• ElasticSearch Service
• Lambda inside a VPC or with the duration of more than 5 minutes
• ECS/EKS/ECR

During these 4 years, I had the opportunity to do a lot of IAM, essential to deploy secure solutions on Amazon.

IAM and least privilege

Identity and Access Management (IAM) is the AWS service that defines users or roles and their associated permissions.

By following the precepts of the AWS well architected framework, the good practice is to assign the finest possible rights for our application to work.

Why this? Several reasons here:

• If my application were to be compromised, I limit the impact as much as possible, because the attacker will only have restricted access to my infrastructure.
• This can allow me to detect abnormal behavior of my infrastructure, if I see a lot of access denied for the role of my EC2 on APIs that it is not supposed to exploit.
• It is a good general hygiene practice in IS to limit as much as possible the perimeter accessible by an application.

The tools provided

In order to position fine rights, AWS allows the creation of policies to indicate the scope of a user or a role.

This tool describes, in JSON format, the rights to be applied. This gives in a very basic way something of this type :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "mapolicyS3",
      "Action": [
        "s3:CreateBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::demo-bucket-tferdinandnet"
    }
  ]
}

As you can see, the policies are human readable and allow for more or less efficient filtering across all API calls and AWS resources.

To assist in the creation of these policies, AWS provides several tools, including :

• Policy generator: https://awspolicygen.s3.amazonaws.com/policygen.html
• Policy simulator: https://policysim.aws.amazon.com/ (only accessible once logged into an AWS account)

In addition, there is documentation of the APIs, with their filtering and resource naming scheme on the official website: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_actions-resources-contextkeys.html.

So everything's perfect?

NO! There are indeed a certain number of tools and documentation on AWS, which will answer a good part of the use cases, as long as you don't try to really make the least privilege ...

Why is that?

Limits

First of all, I'm going to talk about AWS limits. There are two types of limits on AWS:

• Soft limits: these limits are mainly there to protect the customer from misusing AWS and making his bill explode, and can be changed with a support ticket.
• Hard limits: These limits apply identically to all AWS accounts and cannot be modified under any circumstances.

It is important to note that the limits on Amazon's MRI can be reached fairly quickly when trying to severely restrict resource rights, as policies quickly become very verbose.

As an example, you can put a maximum of 10 managed policies on a role or user, with a limit of 6144 characters for each one (without counting spaces), to give an example of the least privilege on a "strong" service like RDS can easily exceed 10,000 characters...

For more information on the limits, I invite you to consult the associated page: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html.

Heterogeneity

If limits were the only problem... We can also talk about the heterogeneity of resource filtering, some are based on the ARN, some on the name of the resource, some on an ID generated at the time of its creation... Concerning this case, it is almost impossible to manage restricted privileges, because to restrict the right to create/modify the resource, it must already exist!

Also, the available conditions are not always the same, or even do not have the same names. A simple filtering on a tag (quite basic on AWS) may not be possible depending on the resources, or the API call made.

The support is out...

Because of the problems I have mentioned above, I had to contact support several times for anomalies that I could not explain.

Apart from the fact that I've been told very frequently "Yeah, your policies are complicated", in most cases, support is useless at best. At worst, it meets my need, not understanding that my job is to secure deployments and infrastructure in AWS...

The support is the champion of the wildcard (*), a magical tool that you wield when you can't find it. The support solution is very often to pass the API call as a wildcard resource or the API itself.

The concern is that from then on, we don't make any more minimum privilege, we certainly restrict, but very far from the minimum...

I could also talk about the tickets I've been opening lately. They were once again known bugs in the IAM AWS that could only be circumvented by wildcard, without any resolution date being given to us .

In conclusion

You'd think with my last paragraphs that I would find AWS's IAM completely off the mark, but that's not the case.

I'm aware that resource filtering is a complex thing to understand and implement, and Amazon is far from being a bad student at this point.

However, I think Amazon would win by giving real examples of least privilege in their documentation instead of inserting wildcards everywhere. It also creates tensions when people like me want to restrict rights since many users don't understand why we are moving away from AWS documentation.

Moreover, the limits on the IAM are aberrant and unnecessary, they just push to give wider rights to save precious characters and make their automation more complex.

Security is a major issue for companies, let's hope AWS hears them!

Traefik is a reverse proxy that we have already mentioned on this blog in the past. Very powerful coupled with containers, it allows a fine and light management of traffic.

A few days ago, Containous, the editor of Traefik, announced the release of Traefik 2.3.0-rc2. This new version brings some changes, including :

• The addition of a new service: Traefik Pilot.
• The ability to add plugins to Traefik
• The addition of the ECS provider

I have already covered the first two points on this blog and I will focus here on the support of the ECS (Elastic Container Service) backend on AWS via a new Traefik provider.

Disclaimer : This post is a translated version of the blog post I made for my company, you can find the french version here, on WeScale blog.

Traefik in the land of providers

What's a provider?

Like Terraform, Traefik uses a notion of provider to define the services it will connect to.

Each provider has its own vocabulary and configuration. The basic idea is of course to have a light kernel, Traefik, and to load only the providers we use.

There are now about fifteen providers available for Traefik, such as Docker, Kubernetes, Rancher, Etcd, Consul etc...

The provider is the data source that Traefik will use to discover the backends it will connect to.

Why the addition of the ECS provider changes the game

A bit of context: ECS is the AWS managed orchestrator, it allows to drive containers on EC2 or on another service, Fargate, which allows to run its containers in serverless mode.

In Fargate, we simply reserve resources, and Amazon takes care of the underlying infrastructure for us (because serverless is not magic).

The addition of the ECS provider allows Traefik to dynamically discover ECS driven resources, in order to attach them directly to itself, which gives more dynamism in your deployments. This discovery is based on Traefik itself using polling via AWS APIs.

Moreover, it allows not to have one AWS ALB per resource, or with a lot of rules, but just one that sends back to Traefik, and it is the latter that takes care of all the routing. Enough to interconnect Traefik with ECS wherever it is deployed. The example here reuses ECS for simplicity, it is not necessary, you can host it wherever you want.

The provider in action

Disclaimer

I therefore propose a small hands-on using this provider, with a small disclaimer:

• This hands-on is made on a "Release candidate" version, so the final version can be slightly different.
• It is realized with Terraform, which is absolutely not a prerequisite, you can achieve the same result with CloudFormation for example
• A bug is currently present in the metadata management on ECS, hence the fact that we go through an API Key, which is not a good security practice in AWS.
• This hands-on is a demonstration, in a productive environment, we will implement stronger hardening on some parameters

What we are going to deploy

Description of the deployment

The complete Terraform deployment code is available here.

I won't describe here the whole Terraform deployment, as it's not the heart of this article, I'll rather focus on the points we are interested in for Traefik, and the nuances to take into account.

The IAM policy, also used as an IAM user in the example, because of the bug described above :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TraefikECSReadAccess",
            "Effect": "Allow",
            "Action": [
                "ecs:ListClusters",
                "ecs:DescribeClusters",
                "ecs:ListTasks",
                "ecs:DescribeTasks",
                "ecs:DescribeContainerInstances",
                "ecs:DescribeTaskDefinition",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

As can be noted, these are read-only rights on ECS and EC2 services, as ECS can exploit EC2 for its execution nodes.

Note that it is quite possible to restrict this policy, in a least privilege model, as recommended by AWS, for example by reducing the scope of ECS visible by Traefik, or by allowing only EC2 with a specific tag.

The policy indicated here is taken from the official documentation.

Deployment prerequisites

Due to the bug with retrieving role information, it is necessary to create a user and assign him an AWS API key and the policy described above. This user is not provided in the code available on GitLab because Terraform does not allow the secret access key to be stored securely. However, if you still want to create this user via Terraform, the following code is required:

/*
Workaround for issue Traefik#7096 : https://github.com/containous/traefik/issues/7096
*/
 
resource "aws_iam_user" "traefik" {
  name = "traefik"
  path = "/system/"
}
 
resource "aws_iam_access_key" "traefik" {
  user = aws_iam_user.traefik.name
}
 
data "aws_iam_policy_document" "traefik_user" {
  statement {
    sid = "main"
 
    actions = [
      "ecs:ListClusters",
      "ecs:DescribeClusters",
      "ecs:ListTasks",
      "ecs:DescribeTasks",
      "ecs:DescribeContainerInstances",
      "ecs:DescribeTaskDefinition",
      "ec2:DescribeInstances"
    ]
 
    resources = [
      "*",
    ]
  }
}
 
resource "aws_iam_user_policy" "traefik_user" {
  name   = "traefik_user"
  user   = aws_iam_user.traefik.name
  policy = data.aws_iam_policy_document.traefik_user.json
}
 
/*Store access keys in Secret manager to retrieve it with Fargate*/
resource "aws_secretsmanager_secret" "traefik_secret_access_key" {
  name        = "traefik-secret_access_key_value"
  description = "contains traefik secret access key"
}
 
resource "aws_secretsmanager_secret_version" "key" {
  secret_id     = aws_secretsmanager_secret.traefik_secret_access_key.id
  secret_string = aws_iam_access_key.traefik.secret
}
 
output "access_key" {
  value = aws_iam_access_key.traefik.id
}
 
output "secret_id" {
  value = aws_secretsmanager_secret.traefik_secret_access_key.id
}

The settings of the ECS tasks

Traefik's task force is quite simple:

[
    {
      "name": "traefik",
      "image": "traefik:v2.3.0-rc2",
      "entryPoint": ["traefik", "--providers.ecs.clusters", "${ecs_cluster_name}", "--log.level", "DEBUG", "--providers.ecs.region", "${region}", "--api.insecure"],
      "essential": true,
      "logConfiguration":{
        "logDriver": "awslogs",
        "options": {
            "awslogs-group": "${loggroup}",
            "awslogs-region": "${region}",
            "awslogs-stream-prefix": "traefik"
        }
      },
      "Environment" : [{
        "name": "AWS_ACCESS_KEY_ID",
        "value": "${aws_access_key}"
      }],
      "Secrets" :[{
        "name": "AWS_SECRET_ACCESS_KEY",
        "valuefrom": "${secret_arn}"
      }],
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80
        },
        {
          "containerPort": 8080,
          "hostPort": 8080
        }
      ]
    }
  ]

The important information here is of course the entrypoint. The entrypoint is the command that will be executed by ECS when the container is launched.

So when we look at the command line parameters we see:

• traefik : The name of the binary to launch, mandatory, since I replace the existing entrypoint in the image.
• --providers.ecs.clusters=${ecs_cluster_name} : This is to name the name of the ECS cluster on which Traefik must search for resources, it is possible to invoke this parameter multiple times, or to tell Traefik to search in all clusters with "--providers.ecs.autoDiscoverClusters=true".
• --providers.ecs.region=${region} : Although it is not specified in the documentation, this parameter is mandatory in order to use access key authentication.
• --api.insecure : Purely optional parameter, it allows access to the Traefik dashboard without the need for authentication, on a productive environment, this parameter is of course not activated.

In addition, we can notice, in environment variables :

• The access key is loaded in plain text, this information is not sensitive, so it doesn't need to be loaded from secrets manager.
• The secret access key, on the other hand, is loaded from secrets manager, so that it is not visible, although in a "secret" section, it is still loaded as an environment variable, but invisible since the definition of the ECS task.

I chose to do the configuration via the command line, but it's quite possible to do it via the Traefik configuration files, for more information about this, I invite you to have a look at the official documentation.

The Whoami taskdefinition, the backend is intended to expose the information used by Traefik :

[
    {
      "name": "whoami",
      "image": "containous/whoami:v1.5.0",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80
        }
      ],
      "dockerLabels": 
        {
          "traefik.http.routers.whoami.rule": "Host(`${alb_endpoint}`)",
          "traefik.enable": "true"
        }
      
    }
  ]

Whoami is a minimalist image, created by Containous for demonstration purposes. It does not require any particular parameters.

You can see that I add two parameters, via Docker labels on my container :

• "traefik.enable: true" indicates that I'm asking Traefik to reference this service.
• "traefik.http.routers.whoami.rule" indicates that I want to create a Traefik "router" called whoami with a rule to forward all traffic that passes through Traefik. These rules are dynamic and can be more complex, once again, feel free to see the official documentation.

Let's deploy our infrastructure

It is now possible to deploy our Terraform code, which will set up our ECS cluster, containers, IAM management and a load balancer front end.

After a few minutes, our Traefik server is now available.

You can access the Traefik dashboard via the url of your load balancer (returned by Terraform), on port 8080.

When you go to the services, you normally see a whoami service, which corresponds to our deployment.

As you can see on the screenshot below:

• The provider is ECS
• We see our "rule" that we had set on the container
• The 3 deployed containers are clearly visible in the backend.

An access to the load balancer url returns the whoami container, when refreshing the page, you should see the IP change, which means that we have load balancing on our service.

To conclude

Traefik adds a new string to its bow by enabling native discovery of services deployed in ECS.

At the moment, you can feel that the varnish is not yet dry, due to the bug that I've reported, but also due to the fact that the documentation lacks clarity on some aspects. So, I do not recommend you to use this feature in its current state on a production environment, again it is a release candidate, we prefer to wait for the stable version for this use.

Nevertheless, this french tech product proves once again that it is able to evolve and adapt to the needs of its users. No doubt about the fact that ECS will soon be a provider among others for Traefik.

Traefik 2.3 (codename: Picodon - picodon is a cheese, which you can see in the banner of this article) is available as a release candidate since a few days. More than a simple version increment, it brings a lot of new features.

Two big new features caught my attention:

• The new service of Traefik : Traefik Pilot
• Adding plugin management

Another new feature, compatibility with ECS will be covered in a future article.

Is there a pilot in the plane?

Traefik is a complete and powerful reverse proxy, as I already presented in previous articles. Nevertheless, it lacked a healthcheck solution.

It is now possible, for free!

A new service is dedicated to this feature, proposed by Containous (the editor, among others, of Traefik).

This service is called Traefik Pilot, it is for the moment very basic and only requires the healthcheck of your Traefik, allowing you to send a notification if it does not answer anymore. It is available at the url https://pilot.traefik.io/

After a quick registration, it is possible to register a Traefik instance.

So I added the command line parameter in the Traefik startup line on Kubernetes.

After a reboot, you should see your status change to OK.

However, there are a few things to keep in mind:

• This status currently only corresponds to the status of your Traefik container, and does not mean that the backends are functional.
• This healthcheck is sent from your instance, in "heartbeat" mode, and does not necessarily mean that your server is reachable.
• As the signal is sent in heartbeat, it is also possible to monitor instances in the application area.

It is then possible, by clicking on your name at the top right, to define alarms, via webhooks or by e-mail.

Note that it is possible to indicate that you wish to receive security alarms, linked to the discovery of possible CVE on your version of Traefik.

Warm up the plugins

I used for many years the very popular Apache httpd. One of the enormous strengths of this product is undoubtedly its modularity, allowing the community to extend its functionality.

Traefik now allows the use of plugins as well. The list is currently rather small, but I have no doubt that the catalog will grow quickly!

Plugins are written in Go, and it is possible to contribute by following the guide provided by Containous.

For the purpose of this article, I chose the "Block Path" plugin edited by Containous. This plugin allows to dynamically block access to certain pages based on regular expressions.

Blocked pages will directly return a 403 (Forbidden) error.

The interest of this kind of plugins, already existing in most reverse proxies, is to be able to intercept access to certain pages, and thus prevent the backend from receiving any hit.

This allows :

• Not to generate a load (for example in the case of an admin page that could be brute force/DDOS).
• To avoid exposing an undiscovered zero day flaw, since the backend does not receive any requests.

Charger le plugin

The loading of the plugin is done via the static configuration of Traefik. For my part, I loaded it via the command line parameters, since this is how I chose to load my entire configuration.

       args:
       - --providers.kubernetescrd
       - --accesslog=true
       - --accesslog.filepath=/var/log/traefik/access.log
       - --accesslog.fields.headers.defaultmode=keep
       - --entrypoints.web.address=:80
       - --entrypoints.websecure.address=:443
       - --certificatesresolvers.le.acme.email=myawesomemail@mail.com
       - --certificatesresolvers.le.acme.storage=/cert/acme.json
       - --certificatesResolvers.le.acme.httpChallenge.entryPoint=web
       - --experimental.pilot.token=mytoken
       - --experimental.plugins.demo.moduleName=github.com/containous/plugin-blockpath
       - --experimental.plugins.demo.version=v0.1.2

You can see line 12 and 13 the loading of the plugin.

In my command line, "demo" is the name I gave to the plugin (used just after). moduleName thus contains the path to the GitHub repository containing the plugin, version being the Git version to checkout.

Once this configuration is done, it is necessary to restart Traefik.

Configuring the plugin

The plugin then behaves like a classic middleware, if you remember my previous articles, middlewares are components that plug between Traefik and your backend, and can alter the normal behavior. For example, on this page, a middleware allows you to define the necessary security headers.

For example, I have declared a new middleware in my Kubernetes :

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: demo
spec:
  plugin:
    demo:
      regex: ["^/demo-[a-z]{1,5}"]

So we find, in a classic way:

• Line 4: The name of my middleware, which can be different from the name I gave to my plugin.
• Line 6: I tell Traefik that this middleware is a plugin
• Line 7: I indicate the name I gave to my plugin when I loaded it
• Line 8 and following, if needed: I now indicate the configuration of my middleware.

In my example, I indicated to block any access whose path would start with "/demo-" with 1 to 5 lowercase letters.

Load middleware

Now that I have defined my middleware, I have to load it into my IngressRoute.

So I modify my IngressRoute by indicating to load this middleware too. As a reminder, you can define multiple middlewares on your IngressRoutes which will be executed in a row.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-web-ui-tls
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - kind: Rule
    priority: 1
    match: (Host(`www.tferdinand.net`) || Host(`tferdinand.net`)) && PathPrefix(`/`)
    services:
    - name: ghost-tfe-fr
      port: 2368
      helthcheck:
        path: /
        host: tferdinand.net
        intervalSeconds: 10
        timeoutSeconds: 5
    middlewares:
      - name: security
      - name: demo
  tls:
    certResolver: le
    options:
      name: mytlsoption
      namespace: default

You can see the load on line 23, which will be executed after adding the security headers.

Let's test the plugin

Now it's time to test my plugin

You can see that it has the expected behavior, my normal accesses work the same way, the accesses that match the schema I defined are blocked by Traefik directly.

In conclusion: A small step for Containous, a big step for the community.

Adding plugins and Traefik Pilot are clearly previews at the moment, and don't allow to take full advantage of their potential, however, this open modularity will allow the community to extend the basic features without making Traefik core heavy.

It also allows companies to potentially develop their own private plugins and thus adapt Traefik to their needs.

Pilot is an excellent initiative, and I can't wait to see how Containous will turn this trial into a top feature!