FREE LOOK

Ubuntu Getting Started With AWS CLI

Prerequisites

This has been tested on Ubuntu 12.04 and 14.04, but may also work on others.

Steps

Installation

Install the tools with the following commands:

Personal Configuration

If you just want to use this for a single account with manual commands (e.g. not scripts), then perform the following steps. If you want to configure this for multiple accounts and/or scripts, then it is probably best to run the "Setup For Scripts" section instead.

Run the following command and answer the questions

aws configure

When it asks for the default output format, your choices are

json

,

table

, or

text

. Personally, I use

json

Setup For Scripts

If you followed the previous section, then you do not need to run this one.

Now we need to set up our AWS credentials for automatic authentication

mkdir $HOME/.aws
vim $HOME/.aws/config

File contents

[default]
aws_access_key_id = [ID HERE]
aws_secret_access_key = [KEY HERE]
region = eu-west-1


It wont work if you put quotation marks around the values.

The configuration needs to be there, otherwise you need to set the AWS_CONFIG_FILE environment variable to its location e.g.

export AWS_CONFIG_FILE="`echo $HOME`/.aws/config"

Protect the file from other users who have access to the same machine.

chmod 600 $HOME/.aws/config

Testing

You can test its working with a simple command t fetch the regions from AWS:

Now you can use the CLI to transfer files to and from s3 like so:

or the other way around

References

• GitHub - aws/aws-cli - Unable to locate credentials
• Installing aws-cli, the New AWS Command Line Tool

download file now

Ubuntu 12 04 Getting Started With LXC

This tutorial is just going to get you hitting the ground running with LXC to prick your interest. As such there is no heavy manipulation of config files or advanced features.

To install LXC and get inside a container simply run the following commands:

sudo apt-get install lxc -y
sudo lxc-create --template ubuntu --name [CONTAINER NAME]

# Wait a significant period of time...

# Start the container and enter into it
sudo lxc-start -d --name [CONTAINER NAME]
sudo lxc-attach --name [CONTAINER NAME]

# have some fun in the container
# exit back out to the host with the "exit" command

# Stop the container when you are finished
sudo lxc-stop --name [CONTAINER NAME]

You can use lxc-start without the -d switch to start the container and automatically enter it. The username will be ubuntu and the password will also be ubuntu, but I had difficulties exiting properly and recommend always using the -d option shown above.

References

• Ask Ubuntu - What is LXC and how to get started?

download file now

Ubuntu FFMPEG H265 Getting Started

High Efficiency Video Coding (HEVC), which is more commonly known as h265, since it is the successor to H.264, has been out for a while now, and ffmpeg added it on the 12th of February 2014. I have had a little bit of practice with it now, and whilst my first reaction was to be blown away by the initial results of the compression (less than 10% initial file size), I have come to realize that video encoding really is a fine art, so this post will be a "dummies" post on getting started with doing a basic conversion with default settings, and being able to play the result. Readers can then delve deeper into mastering the art in an upcoming post.

[ Initial conversion from h264 (bottom) to h265 (top) ]

Installing FFMPEG

Here is how to install FFMPEG on Ubuntu 14.04

sudo add-apt-repository ppa:samrog131/ppa -y
sudo apt-get update
sudo apt-get install ffmpeg -y

Conversion Command

This is an incredibly simple conversion command that converts the video to h265 and just copies the audio, all bundlded within a Matroska file.

ffmpeg -i $inputFile -c:v libx265 -c:a copy $outputFile.mkv

Play H265

Unfortunately, the default VLC for Ubuntu 14.04 (2.1.5 Rincewind), does not appear to have h265 support. We will need to run these commands to be able to play any h265 videos.

sudo apt-add-repository ppa:strukturag/libde265 -y
sudo apt-get update
sudo apt-get install vlc-plugin-libde265 -y

Summary

H265 can produce amazingly small filesizes with a little reduction in quality, or halve the filesize with no detectable change in quality. This could already be having a massive effect on internet based video. As it stands, 1 out of every 3 bits flowing on the internet is video, mostly in the form of Netflix and Porn. Reducing this to 25%-50% means that not only will these files be able to fit more easily onto your mobile devices, but they will be able to stream there a lot more quickly and cheaply, although we will probably just increase resolutions instead.

It did take a lot longer to convert than Im used to but maybe now I can reclaim some space on my BTRFS RAID 10 NFS. I will now keep two copies of each video, a single 480p x264 for streaming to my tablet, and a 1080p or higher copy in x265 for my desktops.

References

• Wikipedia - High Efficiency Video Coding
• Ask Ubuntu - How to install H.265 / HEVC codec on Ubuntu Linux?

download file now

Ubuntu 12 04 ZFS Getting Started

ZFS is a combined file system and logical volume manager designed by Sun Microsystems. I discovered it in my quest to find out about BTRFS. People are stating that BTRFS is superior to ZFS, but BTRFS is not yet considered stable, so in the meantime I am using ZFS. In this tutorial we will be playing with ZFS in order to demonstrate its snapshot/restore capability.

Installation

Lets install all the packages we need as well as updates to the kernel.

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:zfs-native/stable -y
sudo apt-get update && sudo apt-get install ubuntu-zfs -y

You just updated the kernel with some modules, so you need to reboot.
Now run the following command to ensure that ZFS has been set up.
dmesg | grep ZFS
If everything went ok, you will see output like below. If something went wrong, you will get no output.

Setting Up

Now lets create some virtual block devices on which we will set up the ZFS filesystem. If you have some spare hard drives or SSDs plugged in, then you do not need to do this. Create a directory to put the virtual device(s) in:

VIRT_DEVICE_DIR=/virt-devices
sudo mkdir $VIRT_DEVICE_DIR

Create a 100GB "sparse" image to play with:

sudo dd if=/dev/zero of=$VIRT_DEVICE_DIR/1.img bs=1k count=1 seek=100M

Create a ZFS pool with just this one device

POOL_NAME=vol0
sudo zpool create $POOL_NAME $VIRT_DEVICE_DIR/1.img

Check the status
sudo zpool list

sudo zpool status

Now lets mount this filesystem somewhere so that we can add files to it.

MOUNT_POINT=/mnt/data
DATASET_NAME=data
sudo zfs create -o mountpoint=$MOUNT_POINT $POOL_NAME/$DATASET_NAME

Set the mount point to be owned by ourselves so that we can place files within it.

sudo chown $USER $MOUNT_POINT

Test that the dataset was created with:
sudo zfs list

Playing with Snapshots

Now that we have created our ZFS filesystem, we can take advantage as what I see as ZFSs main feauture, the ability to take instant snapshots.

First create a file to check later.

echo "my data" > $MOUNT_POINT/my-file.txt

To see whether snapshots show in the results of zfslist run:
sudo zpool get listsnapshots $POOL_NAME

By default, the display of snapshots is disabled. The following command enables it:

sudo zpool set listsnapshots=on $POOL_NAME

The following command disables it if you want to change back later:

sudo zpool set listsnapshots=off $POOL_NAME

Now take a snapshot

SNAPSHOT_NAME=snapshot1
sudo zfs snapshot $POOL_NAME/$DATASET_NAME@$SNAPSHOT_NAME

Now you should see the snapshot when you perform a zfs list:
sudo zfs list

Now to test the snapshot works, lets change the contents of the file we created just before taking the snapshot.
echo "data changed" > $MOUNT_POINT/my-file.txt
cat $MOUNT_POINT/my-file.txt

Rollback to the snapshot:

sudo zfs rollback $POOL_NAME/$DATASET_NAME@$SNAPSHOT_NAME

Check the contents of the file:
cat /mnt/data/my-file.txt
If you got the message "my data" instead of "data changed" then everything went successfully! You now have a way to instantly restore your filesystem to points in time.

References

• Create a ZFS volume on Ubuntu
• Youtube - Becoming a ZFS Ninja - Part1
• Ubuntu Wiki - ZPool

download file now