gsallesl.github.io

Using LXD for your development environment, with some improvements

2018-08-14T10:20:00+02:00

I will focus on the creation of a container dedicated to android development. With respect to my previous entry on the topic this setup simplifies the directory sharing between the host and the container.

My updated requirements are:

the dev tools will be installed only on the container
the source code should be accessible both from the host and the guest
we setup a local mirror for the android repo to save space as I am working on several versions/platforms

This time we consider the use of a privileged container to ease the directory sharing process between host and container.

Here is what my setup looks like on a ubuntu system:

Container setup

# - Install and configure lxd (one time operation)
sudo apt-get install lxd
sudo lxd init

# - Create a new environment
# for example ubuntu version 16.04 and I call the container ubuntu-1604
lxc launch ubuntu:16.04 ubuntu-1604

# - Create a directory that will contain the code shared 
# between the host and the containers (one time operation)
mkdir ${HOME}/projects

# - Mount the project folder as the container user (ubuntu by default) home 
lxc config device add ubuntu-1604 projects disk source=/home/projects path=/home/ubuntu

# - Create a directory that will contain the android mirror
# !!! This directory shouldn't be user specific (i.e. have the username is the path)
# !!! as the the container user might be different from the host user
sudo mkdir /home/aosp-mirror

# - Mount the project folder as the container user (ubuntu by default) home 
lxc config device add ubuntu-1604 aosp-mirror disk source=/home/aosp-mirror path=/home/aosp-mirror

# - Make the container a priviledged container
# This is required in order to have the correct uid mapping between your guest container and your host system.
# cf previous blog entry for alternative
lxc config set ubuntu-1604 security.privileged true

# - Restart the container to apply the changes
lxd restart ubuntu-1604

The next steps consist in: cloning the android mirror from the container repo sync a version to work on and use reference to the local mirror to save space

Android mirror setup

# Cloning the mirror
cd /home/aosp-mirror
repo init -u https://android.googlesource.com/mirror/manifest --mirror
repo sync

# Sync a working dir against the local repo.
# Use --reference to link the git refs to the mirror
# (optional) Sync only a branch (-c option). Might be dangerous if you want to commit code later on.
cd ~
mkdir aosp-working-dir
cd !$
repo init --reference=/home/aosp-mirror -u /home/aosp-mirror/platform/manifest.git -b android-5.0.1_r1
repo sync -j$(nproc) -c

One advantage of this lxc setup is that the local git references can be resolved both from the host and the container.

Using LXD for your development environment

2016-04-24T10:20:00+02:00

LXD linux containers provides an interesting solution to isolate project development environments (python, android) from your host system.

The motivations for isolating your project development environments are well known and projects such as Vagrant already provide dev environment on top of Virtualbox, Hyper-V or Docker.

However, if you don't need all the functionalities of Vagrant and just look to get a simple and probably more permanent setup, this blog post is for you. This article describes how to use LXD for wordpress development. In this article we will focus on improving the accessibility to the code.

My requirements for my dev environment are as follow:

My dev tools (e.g.: compilers) must be running fast, if possible without the overhead added by a virtualization solution such as VirtualBox or Qemu.
My code must be accessible on my host: I want to be able to edit my code on my host and to have my changes already commited to my dev environment (e.g.: I don't want to have to copy my code from my host to my dev environment, especially when working on big projects such as the linux kernel or the AOSP framework).

LXD provides interesting features for this use case:

Nearly native performances, or at least performances not as bad as they are when compiling in a virtual machine, thanks to the linux namespace system LXD is build on top of.
Easily accessible code: shared folders between the containers and the host use bind mounts. This architecture can be leveraged to editing the code from your host and only compile from the container.

Here is what my setup looks like on a ubuntu system:

Setup

# - Install and configure lxd (one time operation)
sudo apt-get install lxd
sudo lxd init

# - Create a new environment
# for example ubuntu version 14.04 and I call the container ubuntu-dev
lxc launch ubuntu:14.04 ubuntu-dev

# - Create a user that corresponds to the uid given
# to the containers file (one time operation)

# First lookup the uid that corresponds to the container
sudo ls -la /var/lib/lxd/containers/ubuntu-dev/rootfs
[output] drwxr-xr-x  24 165536 165536 4096 Apr 21 20:26 .
[output] drwxr-xr-x+  4 165536 165536 4096 Apr 21 19:13 ..
[output] drwxr-xr-x   2 165536 165536 4096 Apr 20 10:57 bin
[output] ...

# Create the user using the container files uid (one time operation)
sudo adduser --home=/dev/null --no-create-home --gecos "" --uid 166536 --disabled-password --shell /bin/false lxduser

# - Create a directory that will contain the code shared 
# between the host and the containers (one time operation)
sudo mkdir /home/projects
chown -R lxduser:lxduser /home/projects

# - Mount a filesystem outside of the container as the home 
# for the container user (ubuntu by default)
lxc config device add ubuntu-dev projects disk source=/home/projects path=/home/ubuntu

# - Restart the container to apply the changes
lxd restart ubuntu-dev

# - Change ownership of the files in your containers
lxc exec ubuntu-dev -- chown -R ubuntu:ubuntu /home/ubuntu

# - Mount /home/project, which currently belongs to lxduser
# in your host user (gsalles) home tree so you can edit your code locally
# We are going to use bindfs for this operation:
# we use the mapping feature between the lxd container user (lxduser) 
# and your host user (gsallesl)
sudo mount -t fuse.bindfs /home/projects /home/gsallesl/projects -o map=lxduser/gsallesl:@lxduser/@gsallesl

# An fstab entry looks as follow to make the mount automatic at startup
/home/projects  /home/gsallesl/projects     fuse.bindfs     map=lxduser/gsallesl:@lxduser/@gsallesl   0       0

# (optional) Hide lxduser user from the lightdm login screen on ubuntu:
# http://askubuntu.com/questions/92349/how-do-i-hide-a-particular-user-from-the-login-screen/575390#575390

SSH ProxyCommand Resolver

2015-05-12T10:20:00+02:00

SSH ProxyResolver dynamically determines the ProxyCommand to use for a SSH Host alias based on your current network connection and a configuration you provide.

SSH ProxyCommand configuration parameter provides a great way to nicely access a computer not directly accessible on the internet with a simple command.

This configuration parameter is particularly useful on a laptop SSH configuration to access such a system. You may have to change this parameter when switching between networks:

When connected to the same network as myhost:

# .ssh/config
Host myhost
HostName IPAddress

When proxying via a server facing the internet that can access myhost:

# .ssh/config
Host myhost
ProxyCommand ssh internet_facing_server_IP -W IPAddress:some_port

When using a meeting point on another server, for example when the system facing the internet and on the same network as myhost is down:

On my host:

ssh -NR 2345:localhost:22  internet_facing_server2_IP

On the client:

# .ssh/config
Host myhost
ProxyCommand ssh internet_facing_server2_IP -W localhost:2345

If the system myhost actually hosts some of your code on git bare repositories, you may be constrained to use a single SSH alias and adapt the SSH configuration depending on the network you are connected to so that you can still perform pull and push operations. However, you may want to avoid modifying your configuration many times a day.

To avoid having to make these changes manually, here is a little tool, ProxyCommandResolver (pcr), which will pick up the correct ProxyCommand to use for you. pcr uses a configuration file that make an association between an SSH Host alias, multiple ProxyCommand and your current connection parameters (Wifi access point name for example) and returns the ProxyCommand to be used.

So far, the connection parameters supported by pcr are the default routing interface, your wifi ESSID, your NetworkManager connection name or your current IP address.

The tool consists of:

The corresponding SSH configuration to make use of this script is the following. One of the nice feature of the proxy command parameter is that it allows to take the output of a command as a parameter with the following syntax.

Host foo
ProxyCommand $(/path/to/pcr.py %h) # for example $HOME/.ssh/pcr.py

The configuration elements works as follow: * ProxyHost: correspond to the configuration that apply to a SSH Host alias.

Host: the Host Alias as defined in the SSH configuration
Proxy: Delimits a ProxyCommand and the conditions that lead to this specific ProxyCommand
DefaultNetworkInterfaceName: the name of the default network interface that must be matched
LocalConnectionName: either the ESSID or the name of the connection defined in the NetworkManager.
DefaultNetworkIPAddress: the IP address of your default interface.

A Condition section defines one or multiple conditions that must apply to chose a specific ProxyCommand. A Proxy that have the fallback="yes" attribute is used in case of any of the previous Proxy's Condition(s) worked.

This script is designed to work on Linux and has been tested on Ubuntu LTS only. Use at your own risks!

I hope it will help you to save some of your SSH configuration edition time ;)

Compiling PySide 1.0.8 for IDAPRO 6.5 on Windows 7

2015-05-07T10:20:00+02:00

The Compiling PySide for IDA Pro on Windows page provides very useful instructions. However, as we tried to compile PySide following these instructions, we had to make a few changes to the packaging, shiboken, and PySide repositories to make the PySide plugging compile properly.

First you will notice that the build.py script under packaging/setuptools automatically removes the modules directory. This behavior prevents us to apply the initial patch correctly. We broke down this patchset by repository: our new patches can be applied to each repository/module individually.

We also made two small changes to the shiboken patch and PySide patch to successfully compile the PySide module. The resulting installer under packaging\setuptools\dist should work properly.

The 5 corresponding patches are available bellow: