April 13

Linux: How to Automate Docker Deployments

How to Automate Docker Deployments

11th November 2014

For months I’ve been using Drone to continuously build and deploy Docker applications. This has achieved the desired effect of drastically reducing the time required to get code into production. But it wasn’t until recently that I finally got a handle on my deploy script, which was silently leaking old, untagged Docker images onto my hard drives. Below, I share a method for cleanly upgrading a running container, in the form of a bash script suitable for basic automated deployment setups.

Docker Images vs. Containers

First, let’s cover some important points about Docker that will help explain the script below. In Dockerland, there are images and there are containers. The two are closely related, but distinct. For me, grasping this dichotomy has clarified Docker immensely.

What’s an Image?

An image is an inert, immutable, file that’s essentially a snapshot of a container. Images are created with the build command, and they’ll produce a container when started with run. Images are stored in a Docker registry such as registry.hub.docker.com. Because they can become quite large, images are designed to be composed of layers of other images, allowing a miminal amount of data to be sent when transferring images over the network.

Local images can be listed by running docker images:

REPOSITORY                TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu                    13.10               5e019ab7bf6d        2 months ago        180 MB
ubuntu                    14.04               99ec81b80c55        2 months ago        266 MB
ubuntu                    latest              99ec81b80c55        2 months ago        266 MB
ubuntu                    trusty              99ec81b80c55        2 months ago        266 MB
<none>                    <none>              4ab0d9120985        3 months ago        486.5 MB
...

Some things to note:

  1. IMAGE ID is the first 12 characters of the true identifier for an image. You can create many tags of a given image, but their IDs will all be the same (as above).
  2. VIRTUAL SIZE is virtual because its adding up the sizes of all the distinct underlying layers. This means that the sum of all the values in that column is probably much larger than the disk space used by all of those images.
  3. The value in the REPOSITORY column comes from the -t flag of the docker build command, or from docker tag-ing an existing image. You’re free to tag images using a nomenclature that makes sense to you, but know that docker will use the tag as the registry location in a docker push or docker pull.
  4. The full form of a tag is [REGISTRYHOST/][USERNAME/]NAME[:TAG]. For ubuntu above, REGISTRYHOST is inferred to be registry.hub.docker.com. So if you plan on storing your image called my-application in a registry at docker.example.com, you should tag that image docker.example.com/my-application.
  5. The TAG column is just the [:TAG] part of the full tag. This is unfortunate terminology.
  6. The latest tag is not magical, it’s simply the default tag when you don’t specify a tag.
  7. You can have untagged images only identifiable by their IMAGE IDs. These will get the <none> TAG and REPOSITORY. It’s easy to forget about them.

More info on images is available from the Docker docs.

What’s a container?

To use a programming metaphor, if an image is a class, then a container is an instance of a class—a runtime object. Containers are hopefully why you’re using Docker; they’re lightweight and portable encapsulations of an environment in which to run applications.

View local running containers with docker ps:

CONTAINER ID        IMAGE                               COMMAND                CREATED             STATUS              PORTS                    NAMES
f2ff1af05450        samalba/docker-registry:latest      /bin/sh -c 'exec doc   4 months ago        Up 12 weeks         0.0.0.0:5000->5000/tcp   docker-registry

Here I’m running a dockerized version of the docker registry, so that I have a private place to store my images. Again, some things to note:

  1. Like IMAGE ID, CONTAINER ID is the true identifier for the container. It has the same form, but it identifies a different kind of object.
  2. docker ps only outputs running containers. You can view stopped containers with docker ps -a.
  3. NAMES can be used to identify a started container via the --name flag.

How to avoid image and container buildup?

One of my early frustrations with Docker was the seemingly constant buildup of untagged images and stopped containers. On a handful of occassions this buildup resulted in maxed out hard drives slowing down my laptop or halting my automated build pipeline. Talk about “containers everywhere”!

We can remove all untagged images by combining docker rmi with the recent dangling=true query:

docker images -q --filter "dangling=true" | xargs docker rmi

Docker won’t be able to remove images that are behind existing containers, so you may have to remove stopped containers with docker rm first:

docker rm `docker ps --no-trunc -aq`

These are known pain points with Docker, and may be addressed in future releases. However, with a clear understanding of images and containers, these situations can be avoided with a couple of practices:

  1. Always remove a useless, stopped container with docker rm [CONTAINER_ID].
  2. Always remove the image behind a useless, stopped container with docker rmi [IMAGE_ID].

These are the practices built into the upgrade script below.

Deployment Script

The following script is what I use to upgrade a running Docker container to a newer version. This could go in the deploy part of a drone.yml:

#!/bin/bash
docker pull docker.example.com/my-application:latest
docker stop my-application
docker rm my-application
docker rmi docker.example.com/my-application:current
docker tag docker.example.com/my-application:latest docker.example.com/my-application:current
docker run -d --name my-application docker.example.com/my-application:latest

Let’s step through line by line.

1. Pull latest image

docker pull docker.example.com/my-application:latest

We assume that a more recent image has been built and pushed to a registry. If the image hasn’t been tagged, this will result in two new images in the docker images list: one with <none> tag, and one with latest. They’ll both share the same IMAGE ID. If there’s a previous version of this image tagged with latest, it will be untagged.

2. Stop the running container

docker stop my-application

Stop the running instance of my-application. This assumes that we gave the image that name last time it was run.

3. Remove stopped container

docker rm my-application

Now that the container is stopped, it’s safe to remove it. This is cleanup step #1. Also, without this step, we wouldn’t be able to give the name my-application to a different container.

4. Remove image behind stopped container

docker rmi docker.example.com/my-application:current

Now Docker will let us remove the image behind the container we just stopped and removed. This assumes we’ve previously tagged it with current. We need a tag other than latest to be able to differentiate between the version of the app we’re getting rid of, and the one we’re bringing in.

5. Tag the newly downloaded image

docker tag docker.example.com/my-application:latest docker.example.com/my-application:current

Now that the current tag is nonexistent, we tag the downloaded image with current, so that we can identify it next time around. Until step 1 of our next upgrade, the current and latest tags will have the same IMAGE ID.

6. Run the new container

docker run -d --name my-application docker.example.com/my-application:latest

Run the image, being sure to --name it my-application.

That’s it! A possible improvement would be to keep the previous version around to support a quick rollback script. I’m interested in hearing how others are doing automated Docker deployments, and wonder how tools like Kubernetes implement container upgrades under the hood.

By: Caleb Sotelo

Category: Linux | Comments Off on Linux: How to Automate Docker Deployments
January 19

Linux: Ubuntu Mount/USB

Introduction

This page explains how to use USB drives, like external hard disks and USB flash drives (aka USB sticks, thumb drives, pen drives, etc). The material here also applies to flash cards (like in your digital camera).

USB storage devices have the enormous advantage that for the most part they use a standard set of protocols. Thus, instead of needing individual drivers, as does much computer hardware, a standard driver permits access to the devices, making them very portable and able to easily work on many platforms.

Automounting

Mounting

By default, storage devices that are plugged into the system mount automatically in the /media/<username> directory, open a file browser window for each volume and place an icon on your desktop. The rationale for this slight change of behavior can be found here. If you plug in a usb hard disk with many partitions, all of the partitions will automatically mount. This behaviour may not be what you want; you can configure it as shown below.

If the volumes have labels the icons will be named accordingly. Otherwise, they will be named “disk”, “disk-1” and so on.

Configuring Automounting

To enable or disable automount open a terminal and type: 

dconf-editor

Browse to org.gnome.desktop.media-handling.

The automount key controls whether to automatically mount media. If set to true, Nautilus will automatically mount media such as user-visible hard disks and removable media on start-up and media insertion.

Another key, org.gnome.desktop.media-handling.automount-open, controls whether to automatically open a folder for automounted media.

If set to true, Nautilus will automatically open a folder when media is automounted. This only applies to media where no known x-content type was detected; for media where a known x-content type is detected, the user configurable action will be taken instead. This can be configured as shown below.

Configuring Program Autostart

To control which programs automatically start when you plug in a device, go to System-Settings – Details – Removable Media.

Unmounting/Ejecting

Before you disconnect the device, don’t forget to unmount it. This can be done in one of the following ways:

  • Right-click the desktop icon and select “Unmount” (or in some cases, “Eject”).
  • In the file manager window, click on the “eject” button next to the name of the mounted volume.
  • Right-click the icon in the launcher and select “Unmount”.

Auto-mounting (Ubuntu Server)

By default, disk drives do not auto-mount in Ubuntu Server Edition. If you are looking for a lightweight solution that does not depend on HAL/DBUS, you can install “usbmount”.

Manually Mounting

Using Disks

Disks (the GNOME disk utility) is an application for visually managing disk drives and media. When you run it, you will see a list of your drives, including USB drives. If you click a drive on the list, you can view its details, and you can click the triangle-shaped button (Play button) to mount the drive. (This method works even when the drive does not auto-mount.)

Using mount

Get the Information

Sometimes, devices don’t automount, in which case you should try to manually mount them. First, you must know what device you are dealing with and what filesystem it is formatted with. Most flash drives are FAT16 or FAT32 and most external hard disks are NTFS. Type the following: 

sudo fdisk -l

Find your device in the list. It is probably something like /dev/sdb1.

Create the Mount Point

Now we need to create a mount point for the device. Let’s say we want to call it “external”. You can call it whatever you want, but if you use spaces in the name it gets a little more complicated. Instead, use an underscore to separate words (like “my_external”). Create the mount point: 

sudo mkdir /media/external

Mount the Drive

We can now mount the drive. Let’s say the device is /dev/sdb1, the filesystem is FAT16 or FAT32 (like it is for most USB flash drives), and we want to mount it at /media/external (having already created the mount point): 

sudo mount -t vfat /dev/sdb1 /media/external -o uid=1000,gid=1000,utf8,dmask=027,fmask=137

The options following the “-o” give you ownership of the drive, and the masks allow for extra security for file system permissions. If you don’t use those extra options you may not be able to read and write the drive with your regular username.

Otherwise, if the device is formatted with NTFS, run: 

sudo mount -t ntfs-3g /dev/sdb1 /media/external

Note: You must have the ntfs-3g driver installed.

Unmounting the Drive

When you are finished with the device, don’t forget to unmount the drive before disconnecting it. Assuming /dev/sdb1 is mounted at /media/external, you can either unmount using the device or the mount point: 

sudo umount /dev/sdb1

or: 

sudo umount /media/external

You cannot unmount from the desktop by right-clicking the icon if the drive was manually mounted.

Using pmount

There is a program called pmount available in the repositories which allows unprivileged users to mount drives as if they were using sudo, even without an entry in /etc/fstab. This is perfect for computers that have users without RootSudo access, like public terminals or thin clients.

pmount can be used with the same syntax as mount (but without sudo), or quite simply as follows: 

pmount <device> [ label ]

Example:

  • pmount /dev/sdb1 flash_driveThis will mount the device /dev/sdb1 at /media/flash_drive.

If you leave off the label option, it will mount by default at /media/device.

To unmount the device, use pumount, like so: 

pumount <device>

Example:

  • pumount /dev/sdb1

For more help, see the man pages for pmount and pumount.

The Importance of Unmounting

Before disconnecting devices, you must unmount them first. This is similar to “Safely Remove” in Windows in that the device won’t unmount until data is finished being written to the device, or until other programs are finished using it. This applies to all types of storage devices, including flash drives, flash cards, external hard drives, ipods and other media players, and even remote storage like Samba or NFS shares.

Failure to unmount before disconnecting the device can result in loss of data and/or a corrupted file system. There are no exceptions to this rule. Be safe – unmount your drives before disconnecting them!

Other Useful Commands

To see a list of your USB devices (the vendor and device ID’s), run: 

lsusb

To see all attached storage devices and their partitions, run: 

sudo fdisk -l

To see information about currently mounted systems, simply run: 

mount

Troubleshooting

Presented here are some common problems users encounter.

Interfering services

Two services/programs responsible for automounting might interfere and thereby prevent a successful automount and permission setting.

Example: Activating the Automount function of Nautilus while using pmount will result in read-only permissions for normal users. Either disable Nautilus’ Automount function or deinstall pmount.

Unclean LogFile

If you are mounting drives formatted with NTFS (like most external USB hard disks are), you must first have the ntfs-3g driver installed. This is done automatically in newer versions of Ubuntu. You should also install ntfs-config and enable mounting, which is not done automatically.

When a drive is not Safely Removed from a Windows machine (or a forced shutdown occurs from Windows), you may get an error like this when you plug in your drive: 

$LogFile indicates unclean shutdown (0, 0)
Failed to mount '/dev/sda1': Operation not supported
Mount is denied because NTFS is marked to be in use. Choose one action:

Choice 1: If you have Windows then disconnect the external devices by
clicking on the 'Safely Remove Hardware' icon in the Windows
taskbar then shutdown Windows cleanly.

Choice 2: If you don't have Windows then you can use the 'force' option for
your own responsibility. For example type on the command line:

mount -t ntfs-3g /dev/sda1 /media/sda1/ -o force

The best option is Choice 1, but you can force the mount by running Choice 2 with sudo. You must then manually unmount it from the terminal (you can’t right click the desktop icon): 

sudo umount <mount_point>

After that the drive should automount normally again.

User Privileges

If your usb device doesn’t appear on your desktop, you should check that your user has the correct privileges. Go to System->Administration->User and Groups, choose the user, click on “Properties”, then go to the “User Privileges” tab. You should have the “Access external storage devices automatically” option checked.

Preferences

If your usb device doesn’t appear on your desktop, you should check that the automount action is enabled in the preferences:

  • Navigate to System->Preferences->Removable Drives and Media

  • Verify that all “Mount removable drives when…” are checked.

NOTE: This does not seem to apply to Hardy Heron.

USB 2 Issues

old kernels workaround

If you encounter problems using your USB device with USB 2 (i.e. ‘high speed’ mode), you can revert to the ‘full speed’ mode (slower) by unloading ehci_hcd. To do that, type in a terminal: 

sudo rmmod ehci_hcd

before plugging in your device.

recent kernels workaround, from Karmic

ehci_hcd is now built into the kernel and cannot be load/unloaded using modprobe. To revert a connected device from (failing) high-speed to full-speed:

  • Determine your device id using 
     lsusb

     

  • Find which bus it is connected to. The bus id can be found as a folder in /sys/bus/pci/drivers/ehci_hcd. The following script explores buses and connected devices: 
    pushd /sys/bus/pci/drivers/ehci_hcd > /dev/null
for bus in 0000:??:??.? ; do
        echo "ehci_hcd bus $bus"
        pushd $bus/usb1 > /dev/null
        for dev in ?-?; do
                idVendor=`cat $dev/idVendor`
                idProduct=`cat $dev/idProduct`
                echo "ehci_hcd bus $bus: device $dev = $idVendor:$idProduct"
        done
        popd > /dev/null
done
popd > /dev/null

    The information is also usually available in /var/log/kern.log

  • Unbind the bus (and all devices) from the ehci_hcd driver. Insert the bus id in the following command, using the format 0000:00:xx.x 
     sudo sh -c 'echo -n "0000:00:xx.x" > unbind'

     

Buffer I/O Errors

If you see errors related to Buffer I/O when attaching a USB storage device, there are two ways to work around it. First, try using varying max_sectors settings, as such: 

sudo sh -c "echo 120 > /sys/block/sda/queue/max_sectors_kb"

Try values of 120, 64 and 32.

If this does not resolve the issue, then you may need an unusual_dev entry for your device. It would look something like this: 

UNUSUAL_DEV(0x03eb , 0x2002, 0x0100, 0x9999,
            "Generic",
            "MusicDrive",
            US_SC_DEVICE, US_PR_DEVICE, NULL,
            US_FL_IGNORE_RESIDUE),

The vendor and device IDs can be obtained from the output of “lsusb”. The entry would be placed in drivers/usb/storage/unusual_devs.h. If you cannot compile your own kernel, please file a bug report, and we’ll attempt to compile a test module for you.

Device suddenly becomes read-only

If your device changes suddenly to read-only mode, and you see this kind of error: 

[17183798.908000] FAT: Filesystem panic (dev sda1)
[17183798.908000]     fat_get_cluster: invalid cluster chain (i_pos 0)
[17183798.908000]     File system has been set read-only

This might be the sign of an unclean device. You should check your device. Try TestingStorageMedia to do so. Or use “Disk Utility” (under System, Administration), find your device, unmount it, check the file system, then mount it again.

USB-Device is or becomes read-only without errors

If you see “Write Protect is off” and no errors in your logfiles, than you should set filesystem type specific mount options (FS_MOUNTOPTIONS) in /etc/usbmount/usbmount.conf. Wrong gid causes mounting read only.

General tip

When you encounter problems with USB devices, the first thing to do is to check the latest debug information generated from the kernel just after you plug in your device and/or just after you encounter the problem.
To do that, open a terminal and type : 

dmesg

Check the latest messages; they should be related to your problem.

By: Ubuntu

Category: Linux | Comments Off on Linux: Ubuntu Mount/USB
January 11

Linux: Mount Vmware .vmdk file on linux using vmware-mount

Do you have a large .vmdk file on an external HDD that you need to access to pick up some files and don’t want to mount it with  a virtual machine player? 

This quick tip assumes that you are familiar with virtualization and have basic Linux knowledge, but even if you are not, go on and read it, it is basic level and easy to implement.

Vmware Converter, to those who are not familiar with it,  is a tool produced by Vmware that will help you convert a physical machine to a VM (virtual machine), it is a very handy tool if you are running a Vmware datacenter, converting a Physical machine to a VM provides you with mobility for the machine that once was very hard to move or transfer, preserve old software that is running on old risky hardware, Moreover a nice feature is that the Hard Disk real size would be the size of the data inside it, i.e a physical machine that has a 1 TB HDD but has only a 70 GB data would have a VMDK file of 70 GB in size, of course i will not go in to the general benefits of virtualization.

Technically the converter converts that whole physical machine in to a software Hard Disk of type .vmdk which can be run on Vmware player, workstation,fusion,  ESXI, virtualbox… etc, but sometimes you just need to keep a backup of certain machine without deploying it on any player, and when the time comes that you need to pickup some data files from the machine without taking it the whole 9 yards and deploying it, here is where the tool that I will use is called Vmware-mount which is a part of the VDDK (Virtual Disk Development Kit) comes in hand.

The Virtual Disk Development Kit ( VDDK) is a collection of C/C++ libraries, code samples, utilities, and documentation to help you create and access VMware virtual disk storage. The VDDK is useful in conjunction with the vSphere API for writing backup and recovery software, or similar applications.(Vmware definition)

I will be showing the steps of how to mount the .vmdk from the linux terminal : I am using Ubuntu Desktop 14.04 LTS

1. VDDK Download:

If you already have an vmware account you can download the VDDK from the following  URL  or you can use this link URL if you wish, I have uploaded it to google drive in case you don’t have permission to download it, you would need to use a browser to download it,  I have chosen the 5.1 version, because when I installed the 5.5 version the tool that will be used wasn’t available, and  don’t worry you don’t need to worry  you can open the 5.5 version  vmdk with it.

2. VDDK installation:

  • The  downloaded VDDK is a gz tarball that can be extract with the following

tar xzvf %path/to/tar/nameoftar

  •  go in to the directory just created in the same directory and you would find the following perl script “vmware-install.pl”, the file is already executable, but in case it is not, use “sudo chmod +x vmware-install.pl “

run by “sudo ./vmware-install.pl”

  • keep the defaults, the tools would be installed in the /usr/bin so you don’t need to go to the directory, you can just open the terminal and run vmware-mount
  • vmware-mount is like a regular mount, so you need to create a mount point/directory for example:”/mnt/vmwaremount”, this directory will be the destination folder in the commands below.
  • The vmdk used in this example is a typical ntfs windows multi-partition HDD

3. The commands that  you need:

vmware-mount -p /path/to/vmdk
This will list the paritions inside the vmdk, so you would know what partition number to choose.

vmware-mount /path/to/vmdk %partition#% /path/to/mountdirectory
example:
vmware-mount /media/saeed/seagatehdd/mountme.vmdk 2 /mnt/vmwaremount
This will mount the second partition in the vmdk in the mounting point “/mnt/vmwaremount”

Now after mounting the vmdk, you can easily go to the mount/point use the regular Linux commands like:

cd /mount/point
for example
cd /mnt/vmwaremount
ls, cd , cp would be handy from here on

or you can use the GUI to go to the mount directory as an alternative

after you have finished and need to un-mount the vmdk in order to eject the external HDD, in order to un-mount cleanly the vmdk ( be careful the the vmdk would be damaged and would need repair in case you disconnect without cleanly un-mounting the mounted partitions), in order to do that you would need to run the below commands, you would use the -d command on all the mounted points, and you would keep checking the status of the mounted HDD using the -L, as long as the return of the -L command is not empty it means that the vmdk still has a mounted point.

vmware-mount -L
this will displays all virtual disks mounted, but not the partitions

vmware-mount -d /mount/point
This will un-mount the partition
for example to un-mount the partition that we mounted above
vmware-mount -d /mnt/vmwaremount

By: S Khodor

Category: Linux | Comments Off on Linux: Mount Vmware .vmdk file on linux using vmware-mount
December 8

Linux: Making a Ruby script run in cron

You need to setup your crontab with rvm e.g:

rvm cron setup

With that rvm sets your environment variables in your crontab file

then you have a crontab file having something similar to this at the top:

PATH="/usr/local/rvm/gems/ruby-1.9.3-p194/bin:/usr/local/rvm/gems/ruby-1.9.3-p194@global/bin:/usr/local/rvm/rubies/ruby-1.9.3-p194/bin:/usr/local/rvm/bin:/usr/local/rvm/gems/ruby-1.9.3-p194/bin:/usr/local/rvm/gems/ruby-1.9.3-p194@global/bin:/usr/local/rvm/rubies/ruby-1.9.3-p194/bin:/usr/local/rvm/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/rvm/gems/ruby-1.9.3-p194@global/"
rvm_env_string='ruby-1.9.3-p194'
rvm_path='/usr/local/rvm'
rvm_ruby_string='ruby-1.9.3-p194'
RUBY_VERSION='ruby-1.9.3-p194'
GEM_HOME='/usr/local/rvm/gems/ruby-1.9.3-p194'
GEM_PATH='/usr/local/rvm/gems/ruby-1.9.3-p194:/usr/local/rvm/gems/ruby-1.9.3-p194@global'
MY_RUBY_HOME='/usr/local/rvm/rubies/ruby-1.9.3-p194'
IRBRC='/usr/local/rvm/rubies/ruby-1.9.3-p194/.irbrc'

Then you can stick your crontask beneath it using crontab -e

By: bjhaid

Category: Linux | Comments Off on Linux: Making a Ruby script run in cron