A few months ago I spent many hours pulling out my hair in a long attempt to get a Subversion repository accessible through Apache 2.0. The main benefit was the ability to browse through your repository using a web browser and the ability to integrate with the Trac Project Management tool.

The first step is to setup apache2, subversion, and libapache2-svn on your server. Login and type:

sudo apt-get install subversion apache2 libapache2-svn

Next restart the server. Type:

sudo /etc/init.d/apache2 restart

Since we will be using authenticating against specific users in the repository, we also need an SSL certificate for encryption. Type:

sudo a2enmod ssl
sudo apache2-ssl-certificate

 
After creating your SSL certificate, we need to enable port 443 on your server. To do this, edit your ‘/etc/apache2/ports.conf’ file and the line ‘Listen 443’. Your file should look like this:

Listen 80
Listen 443

Next you need to create a virtual host file for your server. If you would like more information on Apache 2.0 and virtual hosts, consult this article. Type:

sudo touch /etc/apache2/sites-available/svn.alexkuo.info

Now you will need to edit your new virtual host file and make it similar to the following.

NameVirtualHost *:443
<VirtualHost *:443> 
        ServerAdmin me@alexkuo.info 
        ServerName svn.alexkuo.info


        <Location "/">
                DAV svn
                SVNPath /var/svn/akuo
                AuthType Basic
                AuthName "ALex Kuo's Repository"
                AuthUserFile /etc/apache2/dav_svn.passwd
                <Limit GET PROPFIND OPTIONS REPORT>
                        Require valid-user
                </Limit>
        </Location>

        CustomLog /var/log/apache2/svn.alexkuo.info.custom.log combined
        ErrorLog /var/log/apache2/error.svn.alexkuo.info.log 

SSLEngine on
SSLCertificateFile /etc/apache2/ssl/apache.pem
SSLProtocol all
SSLCipherSuite HIGH:MEDIUM 
</VirtualHost> 

In the first and lines, ‘NameVirtualHost *:443’ and ‘<VirtualHost *:443>’, tells the server to listen for all names on port 443, which is the SSL port that we specified previously. The ‘ServerName’ variable specifies the domain that this configuration applies to, in this case svn.alexkuo.info. The Location tags can be used to specify different references to repositories. In this example, their is only one – ‘http://svn.alexkuo.info/’.

In the location tag, ‘SVNPATH’ specifies the local path to the repository and the ‘AuthType’ variable specifies authentication type. The setting ‘Basic’ will just prompt for a username/password. The AuthUserFile is the authentication file that will be used to authenticate against when a user atttemps to access the repository. The settings in the <Limit> specify the permissions given to a valid-user.

Next, let’s setup a user for our repository. Type:

sudo htpasswd -c /etc/apache2/dav_svn.passwd svnuser

Now we should setup a repository at the location specified by the variable ‘SVNPath’ in our virtualhosts file. We also need to set the owner to ‘www-data’ and give that group read/write permissions to these directories. To do this type:

sudo svnadmin create /var/svn/akuo
sudo chown -R www-data:www-data /var/svn/akuo
sudo chmod -R g+ws /var/svn/akuo 

The last thing we need to do is enable the site and restart Apache. Type:

sudo a2ensite svn.alexkuo.info
sudo /etc/init.d/apache2 restart

You should now be able to access your Subversion repository through the ‘http’ protocol.

Reference Links

• Ubuntuforums.org – HOWTO – Apache2 + Subversion + SSL – Shares a lot of information discussed in this tutorial.
• Official Subversion Book

In a previous article, I described how to deploy the Django based application to a server running Ubuntu Server 7.04 and Apache 2.0 running mod_python. This article intends to expand upon that configuration by adding automated deployment using CruiseControl and Subversion source control server. CruiseControl is a java based server that has the ability to monitor a variety of source control servers, such as Subversion,  for changes and execute scripts or programs for deploying and configuring an application whenever changes are detected in source control.

Installing CruiseControl requires downloading the Java 6 runtime files, downloading the CruiseControl application, and configuring which source control paths to monitor and their associated actions. My CruiseControl configuration consists of two separate ‘projects’ – one for monitoring static files and another for monitoring the Django application itself.

First install the Java 6 runtime and JDK files. You can do this by typing:

sudo apt-get install sun-java6-bin sun-java6-jre sun-java6-jdk

Before downloading the CruiseControl application, I created a separate user account , called ‘deploy’, for monitoring and executing deployments. This created a separate home directory for the user and alleviated any worries about permissions  or ownership that may affect directory or file access.

You can create a new user called ‘deploy’ and add it to the administrative group by typing the following:

sudo adduser deploy
sudo adduser deploy admin

After creating the new user account, log into the new account and download CruiseControl. Type:

wget http://superb-west.dl.sourceforge.net/sourceforge/cruisecontrol/cruisecontrol-bin-2.7.zip

Unzip CruiseControl by typing

unzip cruisecontrol-bin-2.7.zip 

Since this is a java application, the only thing required for installation is a directory for the java files to live in. In my configuration, I renamed the CruiseControl directory from ‘cruisecontrol-bin-2.7’ to ‘cc’. Type:

mv cruisecontrol-bin-2.7 cc

My path for CruiseControl now looks like this: ‘/home/deploy/cc’

If you haven’t yet, you need to add the JAVA_HOME path variable to your account. You can do this by editing the ‘.profile’ file in the current user’s home directory. I did this by typing:

vi ~/.profile

After opening the file, scroll to the very end of the file and append the following text:

JAVA_HOME="/usr/lib/jvm/java-6-sun-1.6.0.00/jre"

export JAVA_HOME 

This will add the JAVA_HOME path variable. The variable should be set to the location of the java-6 jre subdirectory. In order for these changes to take place, you need to re-login into your account.

After logging into your account again, go to the directory where you extracted CruiseControl and run the CruiseControl shell script. I did this by typing:

sh cruisecontrol.sh

This initiates CruiseControl for the first time and allows the server to test itself against an example project that comes with the default installation. If you see a few errors related to accessing an Subversion repository, don’t worry about it.

After the confirming that the server works, stop the server by killing the process. I did this by finding the pid in the ‘/home/deploy/cc/cc.pid’ file. Type:

cat cc.pid
sudo kill PID_NUMBER

In the following text, we’re going to configure CruiseControl to deploy a Django application. First, let’s remove the example project (which is called ‘connectfour’) and replace it with a Django deployment project and a project for deploying static files. In the previous article, I created two separate deployment locations – one called ‘assets’, which stores static files, and another called ‘ALEXKUO’, which is the Django Application.

The CruiseControl project names will be ‘assets’ and ‘webapp’. In your CruiseControl directory [‘/home/deploy/cc’], type:

cd projects
rm -rf connectfour
mkdir assets
mkdir webapp

The projects directory will contain the Python scripts used to delete and re-checkout files from Subversion. We also need to setup the logs directory. In your CruiseControl directory [‘/home/deploy/cc’], type:

cd logs
rm -rf connectfour
mkdir assets
mkdir webapp

Next we need to edit the CruiseControl configuration file or ‘/home/deploy/cc/config.xml’. To open type:

vi config.xml

My configuration file looks like the following:

<cruisecontrol>
<project name="webapp">

  <bootstrappers>

    <currentbuildstatusbootstrapper file="logs/webapp/buildstatus.txt"/>

  </bootstrappers>


  <modificationset quietperiod="10">

    <svn RepositoryLocation="http://svn.alexkuo.info/trunk/ALEXKUO" />

  </modificationset>


  <schedule interval="30">

    <exec workingdir="/home/deploy/cc/projects/webapp"

    command="python"

    args="deploy.py" />

  </schedule>


  <listeners>

    <currentbuildstatuslistener file="logs/webapp/buildstatus.txt"/>

  </listeners>


  <publishers>

    <currentbuildstatuspublisher file="logs/webapp/buildstatus.txt"/>

  </publishers>

</project>

<project name="assets">

  <bootstrappers>

    <currentbuildstatusbootstrapper file="logs/assets/buildstatus.txt"/>

  </bootstrappers>


  <modificationset quietperiod="10">

    <svn RepositoryLocation="http://svn.alexkuo.info/trunk/assets" />

  </modificationset>


  <schedule interval="30">

    <exec workingdir="/home/deploy/cc/projects/assets"

    command="python"

    args="deploy.py" />

  </schedule>


  <listeners>

    <currentbuildstatuslistener file="logs/assets/buildstatus.txt"/>

  </listeners>


  <publishers>

    <currentbuildstatuspublisher file="logs/assets/buildstatus.txt"/>

  </publishers>

</project>
</cruisecontrol>

As you can see from my config.xml file, the project tag defines a unique project. Each project monitors a specific repository location for changes. These repository locations are defined by the ‘modificationset’ and ‘svn’ tags. Whenever changes are detected, the schedule and exec tags are activated. The publishers, listeners, and bootstrappers files are log files that record activity during monitoring, building, and deployment.

The ‘deploy.py’ files contains a python script for deleting a target directory and re-checking out files from a target SVN location. Let’s create these files.

For the webapp project, create a file call ‘deploy.py’ in the ‘/home/deploy/cc/projects/webapp’ directory. I did this by typing:

touch /home/deploy/cc/projects/webapp/deploy.py

Next, let’s edit this file. In the ‘/home/deploy/cc/projects/webapp’, type:

vi deploy.py

My deploy.py for the webapp looks like this:

from os import system, fchdir, getcwd
from shutil import copy, rmtree

import os

DJANGO_APP_DIR = '/home/deploy/django/django_projects/ALEXKUO'

SVN_ROOT = 'http://svn.alexkuo.info/trunk/'
SVN_APP = 'ALEXKUO'

APP_SETTINGS = '/home/deploy/django/appsettings/ALEXKUO/settings.py'

CMD_SVN_CHECKOUT = 'svn co ' + SVN_ROOT

cout_app = CMD_SVN_CHECKOUT + SVN_APP + DJANGO_APP_DIR

IS_APP_UP = False

"Remove all files in App directory and check out files"
rmtree(DJANGO_APP_DIR)
system(cout_app)
if os.path.exists(DJANGO_APP_DIR):
    IS_APP_UP = True

if IS_APP_UP:
    copy(APP_SETTINGS, DJANGO_APP_DIR)

The script first removes all files in the targeted Django application directory, this is done with the command ‘rmtree(DJANGO_APP_DIR)’.  I also copy a ‘settings.py’ modified for the server because the Django configuration between my local development box and public server is different. The SVN checkout is initiated with the command ‘system(cout_app)’.

Next let’s create the deploy script for the asset files. In the ‘/home/deploy/cc/projects/assets’ directory, type:

touch deploy.py

Now let’s edit this file by typing:

vi deploy.py

My deploy.py for the assets project looks like this:

from os import system, fchdir, getcwd
from shutil import copy, rmtree

import os

ASSET_DIR = '/home/deploy/websites/www.alexkuo.info/assets'

SVN_ROOT = 'http://svn.alexkuo.info/trunk/'
SVN_ASSET = 'Assets '


CMD_SVN_CHECKOUT = 'svn co ' + SVN_ROOT

cout_asset = CMD_SVN_CHECKOUT + SVN_ASSET + ASSET_DIR

"""Delete Asset directory and re checkout asset folder """
if os.path.exists(ASSET_DIR):
    rmtree(ASSET_DIR)

system(cout_asset)

This python script is identical to the deploy script for the webapp, except that it excludes commands for copying the ‘settings.py’ file.

After creating your python deployment scripts, your build server should be ready for use. Turn on the server by going to the cruise control directory (/home/deploy/cc) and typing:

sh cruisecontrol.sh

You can now log off the account and your server will still be running. You can access the server’s status page by opening your browser and typing in the URL: ‘http://<server’s IP address>:8080/”

Reference

• Subversion Home
• CruiseControl Home
• CruiseControl Documentation: Getting Started

For web developers who don’t have much system administration experience, deploying applications to production is the last hurdle that just doesn’t happen smoothly either because of a lack of experience or because of a lack of planning beforehand. The following is a how to guide based on my first experience of deploying a Django application to an environment running the following:

• Ubuntu Server 7.04 Fiesty
• Apache 2
• mod_python
• Django Framework
• The Python database adapter or API that your application uses. In my case, it was ‘psycopg’.
• The Database your application uses. I’m using PostgreSQL 8.2 for this deployment

Other things you will need in order to a install Django application include:

• Subversion 1.2+
• FTP Client
• Administrative access (‘sudo’).
• An application using Django Framework
• Access to the virtual host files in the /etc/apache2/sites-available’
• A website setup using Apache 2.0

I assume the user already has Ubuntu, Apache 2, and a database installed on the target host. If you haven’t already installed mod_python for Apache 2 and Subversion, you can do so by SSH’ing or logging into your server and type the following:

sudo apt-get install libapache2-mod-python subversion

Next we need to setup the directories that will be holding our Django project files and the Django framework. I usually place these files in a subdirectory in the user’s home directory. For my last deployment, I made a root directory called DJango with two subdirectories: django_src (stores framework) and django_projects (stores your django applications). I did this by typing:

mkdir /home/alex/django
mkdir /home/alex/django/django_src
mkdir /home/alex/django/django_projects

We should now download the Django framework. We can get the latest copy using Subversion. I did this by typing:

sudo svn co http://code.djangoproject.com/svn/django/trunk /home/alex/django/django_src

Next, we need to add the Django framework to the Python interpreter. I did this by adding a symbolic link to Python’s site-packages directory. In this environment, I am using Python 2.5. Type:

sudo ln -s /home/alex/django/django_src/django /usr/lib/python2.5/site-packages/django

In the next following paragraphs, I’m going to go into detail on how I organized and setup my Django application files on the server. When developing a Django application, there are two separate types of files being deploy: static files that are not processed by the Apache mod_python module, such as CSS, Django templates, javascript, and images, and Python files that need to be processed by the Apache mod_python module (.py files).

I prefixed all references to static files with ‘/assets/’. This is going to be the root directory in our website that will hold all static files. On a side note, your source control directories should have separate directories for holding Django python files and static files used by your Django application.

In the target website folder on the host, I created a subdirectory called ‘/assets/’. I did this by typing:

sudo mkdir /home/alex/websites/www.alexkuo.info/assets

Copy your static files into the assets directory.

Django applications commonly use the admin tool. In my projects, I specify the path to the static admin tool files by adding a symbolic link called ‘/admin_media/’. You can do this by typing:

sudo ln -s /home/alex/django/django_src/django/contrib/admin/media /home/alex/websites/www.alexkuo.info/admin_media

Next we are going to copy your Django application files to the server. To do this, create a Django folder named after your application. It is important that you name the directory the same as your application name. As a result, the name of the directory will be case sensitive. To find out your application’s name, open the ‘settings.py’ file and look for the variable ‘ROOT_URLCONF’. The application name can be found by looking at its value. For example, ‘APPLICATION_NAME.urls’.

I created the folder by typing:

mkdir /home/alex/django/django_projects/your_django_project_application_name

Before copying your application files over, I recommend editing the settings.py so that it will work properly with your host environment. The variables that need to be set that are specific to deployments are :

• Database variables
• MEDIA_ROOT
• MEDIA_URL
• ADMIN_MEDIA_PREFIX
• TEMPLATE_DIRS

 The following is a portion of my settings.py file:

....
....
....

DATABASE_ENGINE = 'postgresql'

DATABASE_NAME = 'alexkuoinfo'
DATABASE_USER = 'alex'
DATABASE_PASSWORD = '******'
DATABASE_HOST = 'db.alexkuo.info'

DATABASE_PORT = '5432'

MEDIA_ROOT = '/home/alex/websites/www.alexkuo.info/assets/'

MEDIA_URL = 'http://www.alexkuo.info/assets/'

ADMIN_MEDIA_PREFIX = '/admin_media/'

TEMPLATE_DIRS = (

    "/home/alex/websites/www.alexkuo.info/assets",

)

.....
.....
.....

Copy your django application files to ‘/home/user/django/django_projects/your_django_application_name’.

Now we need to setup Apache to process requests using mod_python. This requires the user to edit the virtual host file for the site. On my server, I am using a unique virtual host file for each site. These files are located in the ‘/etc/apache2/sites-available/’ directory. If you would like more information on adding virtual hosts to your Apache installation, consult this tutorial.

Edit the your virtual hosts file for the site, to do this I typed:

cd /etc/apache2/sites-available
sudo vi www.alexkuo.info

After editing my virtual host file, it looked like this:

<VirtualHost *> 
        ServerAdmin me@alexkuo.info 
        ServerName www.alexkuo.info

        DocumentRoot /home/alex/websites/www.alexkuo.info/ 

	<Location "/"> 
	        SetHandler python-program 
	        PythonHandler django.core.handlers.modpython 
	        SetEnv DJANGO_SETTINGS_MODULE ALEXKUO.settings 
	        PythonDebug On 
	        PythonPath "['/home/webdeploy/django/django_projects'] + ['/home/webdeploy/django/django_projects/ALEXKUO'] + sys.path" 
	</Location> 
	<Location "/admin_media/"> 
		SetHandler None 
	</Location> 
	<Location "/assets/">
		SetHandler None
	</Location>
        ErrorLog /var/log/apache2/error.www.alexkuo.info.log 
</VirtualHost> 

After saving your changes to the virtual host file, you should restart Apache. Type:

sudo a2ensite www.alexkuo.info
sudo /etc/init.d/apache2 restart

The settings in the virtual host file unique to a Django application are the Location elements. The element ‘<Location “/”>….</Location> ‘ tells Apache that all requests received from the root directory and subdirectories should use mod_python to interpret the request. The environment variable, ‘DJANGO_SETTINGS_MODULE’ specifies the settings module used for Django’s PythonHandler. The PythonPath references your Django application’s directory. You have to specify the root and subdirectory paths for your Django application or else import commands will fail when your application is compiled and run.

The next two Location elements specify that files in “/admin_media/” and “/assets/” directories should not be handled by mod_python.

If you’re working with a development machine and would like to see changes in your application files immediately, it is convenient to force Apache2 to reload your Django application per request. To do this, add the following in your ‘/etc/apache2/httpd.conf’ file and restart the Apache webserver.

MaxRequestsPerChild 1

Another alternative to reloading your Django application is to change the ‘settings.py’ file for your application. This is useful if you have other websites or applications running on your server. I usually just change the DEBUG variable from False to True.

That’s it. Since this is my first deployment, I’m open to any comments, suggestions, questions, or critiques.

For more information about deploying Django applications, please consult the following links:

• Jeffcroft.com – Setting up DJango on DreamHost. He uses fastcgi in this tutorial, but alot of the information is still useful.
• Djangoproject.com – Official documentation using mod_python

Setting up virtual domains in Apache 2.0 is pretty darn easy. Easier then in IIS 6? Well, it requires more typing, but it takes literally less then 5 minutes to get a new domain up on a server.

Before I start, I am making a few assumptions about the operating environment:

• Ubuntu server is being used, along with Apache 2.0.
• The user has control over the targeted domain.
• The user has ‘sudo’ or access to the ‘su’ account.

First, the user needs to point their domain to their server by adding an IP address entry for their domain. For example, this domain has a DNS entry for an IP address pointing to ‘www.alexkuo.info’. Once you have that setup, SSH into your web server.

The first thing I always do is setup a directory to store the files for the website (ex. ‘.html’, .’php’, ‘.jpg’ files). I usually setup a subdirectory in my home directory because I know I already have permissions to read,write, and execute files there.
To do this I type

mkdir websites
cd websites
mkdir www.alexkuo.info

sudo chown www-data:www-data /home/alex/websites

The first three lines create the subdirectory /home/alex/websites/www.alexkuo.info, while the last line sets directory /home/alex/websites and all other subdirectories below it to be readable by the default Apache user.

Next we need to add a virtual host to Apache. You can do this by typing

cd /etc/apache2/sites-available/
sudo touch www.alexkuo.info

The first line navigates to the sites-available directory. Files in here should contain information on the virtual hosts available. Virtual hosts correlate a domain received by Apache and the location of the files to be served for the received domain. The second line creates a file www.alexkuo.info. We will be using this file to configure the virtual host for the domain ‘www.alexkuo.info’.

I usually use ‘vi’ editor to edit text files on servers, and I know ‘vi’ is probably the most user-unfriendly text editor on earth. Instead of using ‘vi’, the user can use any text editor of their choice. In the ‘/etc/apache2/sites-available/’ directory, the user can setup the virtual host ‘www.alexkuo.info’ by typing:

sudo vi www.alexkuo.info

After opening the file, press ‘i’. This will set vi into text editing mode. Afterwards, copy and paste (or type) the following text into the text editor.

<VirtualHost *>
	ServerName www.alexkuo.info

	DocumentRoot /home/alex/websites/www.alexkuo.info/
	ErrorLog /var/log/apache2/error.www.alexkuo.info.log
</VirtualHost>

After the above has been entered, press the [ESC] key and type ‘ZZ’ (or [shift-z][shift-z] in rapid succession). This saves and exits the editor.

This is the most basic configuration you can have for a virtual host. The ‘DocumentRoot’ setting specifies the location of the files to be served when the domain, set by the ‘ServerName’ configuration, is received.

If you haven’t disable the default site, you will need to in order for the server to see the virtual host you created. You can do this by typing:

sudo a2dissite default

After saving the file, you will need to restart Apache2. You need to enable the site and restart the Apache. You can do this by typing the following:

sudo a2ensite www.alexkuo.info
/etc/init.d/apache2 reload

Your site is now setup and live. All that is left is to upload files to your website directory that was created at the beginning of this article. For this website, the directory is ‘/home/alex/websites/www.alexkuo.info/’.

If you would like to setup another site, just repeat the above procedures, except substitute the domain ‘www.alexkuo.info’ with another domain, like ‘subdomain.alexkuo.info’.

If you would like more information on the topic, please consult the following links:

• Ubuntuforums.org – Brief forum post about setting up sub-domains in Apache
• Trac on Ubuntu – Goes into detail setting up the Trac server on Ubuntu, but contains alot of pertinent information used in this tutorial
• Apache 1.3 Docs: Name-based Virtual Hosts – A bit dated but the configuration for Apache 2 is almost the same.

I’ve been messing around with a few programs on Gnome lately and a few of them require the user to execute a file called ‘install.sh’ in the command prompt in order to initialize the installation process. In order to do this you just type:

sudo sh install.sh

- or just -

sh install.sh

Pretty straight forward right? Well… not always. Typically installing a downloaded package requires you to extract your files into a directory. There are two notes that Linux users assume everyone knows:

• You have full read, write, and execution permissions to the directory and files you are extracting.
• You actually know which directories and files have these permissions.

At first this sounds like a no brainer, but to the ex-Windows user who had administrative access to all permissions by default, this represents a new experience. When installing your own programs that are not from ‘apt-get’ or from the Ubuntu Package Manager, I suggest users download new packages and install new programs in subdirectories located under the ‘/home/user_name/’ directories.  As for myself I have directory for new packages called ‘/home/alex/downloaded_packages’

You can view privileges for all your new files and directory by navigating to the location of your extraction and typing:

ls -l

Which typically results in a directory listing that looks like this:

total 6100
-rwxr-xr-x  1 alex         725 Jul 10 15:07 stuff.txt
drwxr-xr-x  2 alex         512 Jun 13 16:19 dev
....

The entries that start with a d are directories. The next 3 letters are permissions for the owner. The owner is specified by the name in the third grouping of characters from the left.

Even after downloading the new package, the default permissions assigned to the extracted files and directories may not have sufficient privileges to install the program after running ‘sh install.sh’, resulting in a permissions error. To remedy this, you need to give all extracted files and directories execution privileges. You can do this by going to the location of the extracted files and typing the following:

find . -type d -exec chmod 700 '{}' ';'
find . -type f -exec chmod 700 '{}' ';'

The first command sets all directories in the current directory permissions to read, write, and execute, while the second command does the same for files. This will give the owner of the files sufficient permissions to run the ‘install.sh’ command.

If the above doesn’t work, you can try giving everyone full rights to the files in the current directory.  A warning though, this should only be done to files that are temporary and only to files that you trust. A lot of security holes in Windows is attributed to the fact that full rights are given to all files during execution. To do the aforementioned type:

find . -type d -exec chmod 777 '{}' ';'
find . -type f -exec chmod 777 '{}' ';'

Hopefully, this should give you the ability to run those ‘install.sh’ files you get. If you’d like more information on the above consult the following:

• The ‘ls’ Command
• The Advanced Bash-Scripting Guide

Today I had to add a new user to a server running Ubuntu Server 7.04 using the Bash shell. The new user’s login will be ‘bob’.

1. First create the user’s login to the system – type:  adduser bob
2. Afterwards, I wanted to add the new user to a few groups, specifically groups I belong to in the system. So I typed: groups alex
3. Now you can add whatever groups you think bob needs to belong to by typing: adduser username groupname

If you would like to know more about administering users, I recommend looking at the following links:

• Oreilly: Managing Users in Linux
• How-to-geek: Groups your linux user belongs to