frontier technologist

Mount Samba With UTF8

2011-05-31T13:56:00.001-04:00

Hey dumbass-

You will reference this link again, I promise.

http://ubuntuforums.org/showthread.php?t=288534

Signed,
Dumbass.

Django "duplicate key value violates unique constraint"

2011-05-31T13:20:00.000-04:00

The few times that I have seen this error in my own code, finding the source and squashing any related bugs was pretty easy. For the first time today, I saw it appear when I was using the provided admin interface to add a user to my project where I was using django.contrib.auth. It caused me to panic a little.

With a little bit of google searching, I found this post on the django google group that helped me fix my problem clearly and quickly.

I am using a postgres back end, and it seems like the counter for the serial field was reset. By looking at the table and finding which value should come next, I was able to set this counter to the proper value, and so far everything is working great.

VirtualLab

2011-05-30T21:57:00.001-04:00

I'm going to begin documenting a project with the tentative title of VirtualLab.

VirtualLab will attempt to be a complete test-bench of environment of heterogeneous clients and servers interacting.

In addition to being a tool with a real-world use, the VirtualLab project is also meant to provide an educational tool. The VirtualLab project aims to teach the ability to bootstrap itself.

I'm trying to carve out the first piece of this project now, is to start by selecting and implementing the bare metal foundation. A virtualization stack needs to be selected from free and license-compatible options. The initial list is KVM, Xen, VMWare and Oracle.

Crossbeam

2011-05-25T07:34:00.000-04:00

Trying to find a name for the project, and this is the first dart I'm throwing at it. Crossbeam for Coarse Search for Barcodes (CSB). It's the best thing that
grep -i '^c.*s.*b' /usr/share/dict/words
turned up. It's kind of evocative of physical laser barcode scanners so it could work.

Anyway, the first part will be the module that actually interacts with output. I need to write a layer where I convert PDF output into images that can be processed, and that part will be taken care of by a system call to the ImageMagick program convert.

Coarse Search For Barcodes In A Large Images

2011-05-25T00:33:00.001-04:00

There are a few decent python modules that are capable of reading barcodes in images. They work reliably if the barcode is large enough in the image. They do not provide any kind of ability for searching a large image for a small barcode (that I know of). I require a module that guesses coordinates until barcode(s) are extracted from an image.

The first part of this project is some scaffolding. The project should have an interface between a search algorithm and a scaffold that tries search results and parses the results of trying those search results. The project is then divided into two parts, on either side of that interface. This interface is meant to allow creating pluggable components by completely decoupling the barcode scanning package from the search algorithm.

The motivation to solve this problem comes from extending a document management system. The source images are going to be classes of forms that have barcodes always in the same spot. I see the project as having a significant potential to be useful to lots of people...if done well. I think the most important part is sticking to this interface.

Since the first use of this project will be to scan a certain well-defined form, it will start with a search module that is specifically designed to extract a barcode from that form. It's important to decouple any implementation details of this "dumb" search module from the as-yet undefined abstract search API.

This module will be hard-coded to find one particular barcode most likely on the first guess of a box coordinate. Stated like this, the first definition of the project's API comes into light. We are given a BCS (box coordinate slice) to try, and parse the results. The parser will black box the actual module that performs the barcode analysis. I think a certain constraint needs to be introduced...that a BCS is meant only to contain a single barcode. The user should be easily able to define a regular expression that checks the validity of the barcode. This would be for a convenience layer, that could potentially be an API to a set of document classes. A document class would define a list of barcodes and corresponding optimized search functions. This could be the beginning of a third component of the system, or that be more integrated with the document management system than the actual barcode scanning system.

The creation of the hard-coded search for a particular form is important. Firstly, our goal should be to get something functional as soon as possible. In order to iterate towards this goal as soon as possible, a requirement will be the ability to benchmark the search on a reasonable test data set from the beginning.

The process of creating a hard-coded search module will start with a guess for the BCS that works on the test data set. The document has a pretty gigantic barcode on it, for now, with a goal in mind to shrink it. A BCS of the quadrant of the document that contains the barcode will be the first guess, and I'm not going to change that guess until the project can successfully analyze the effectiveness of that guess, and adheres to the design laid out in this first document.

Video talk: Real Software Engineering

2010-09-10T01:20:00.003-04:00

Real Software Engineering - Glenn Vanderburg

This talk was given at a Ruby conference, but it really has nothing to do with Ruby or Rails at all.

The intro, and partially, the thesis, of the talk is that software engineering has been misunderstood. In particular, he says that theories taught at universities are completely misguided and result in projects that are over budget, behind schedule, or in some cases even complete failures. I studied a little bit of software engineering in university and I was never taught any of the points that he criticizes. The ideas that he says should replace the bad ideas have been outlined in books such as The Mythical Man Month (1975) and Code Complete (1993). If you grok those books and beyond, then this video might be boring to you; however, he concisely explains a few important points that form the core of software engineering.

You can skip the first 15 minutes if you're short on time, but it's at least engaging.

Initially, one of the principals of software engineering was that the cost of changing the project increased near the tail end of a project. This data had a hidden bias, because all of the projects measured followed the flawed waterfall model. Really, the data being measured was more semantically described as the "cost of finding and fixing errors as a function of the distance in time from when the error was actually made (i.e. long feedback loops." The insight to be gained from the data, from this perspective, is that errors need to be rapidly discovered. By interlacing testing into the entire development process, the cost is reduced.
He describes a spectrum of process models defined process model (requires that every piece of work be completely understood; a defined process has the same results every time) at one end and the empirical process model (provides and exercises control through frequent inspection and adaption for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs) at the other. In particular, there's a bit of history where software engineers borrowed the defined process model from chemical engineering, when chemical engineers all knew that no real process is ever completely understood.
Mr. Vanderburg made a very good point using the Boeing 777 as an example. As a test of the wing design, the engineers devised a test where the wings would be subjected to 150% of the maximum force they were ever expected to encounter in use. Video of the test is pretty cool. The point is that engineer is more concerned with practice than theory. Mathematics was originally applied to engineering as a cost saving method. Real-world tests are in many ways more valid than mathematical model validation. A quote I liked: "Ask your friendly neighborhood aerospace engineer how much math he would do if building a model of his design and testing it were effectively instantaneous and free...the answer would probably be 'A lot less than I do now.'" [I paraphrase for clarity a bit at the end.]
He follows up with a treatment of the Structural Engineer's Association definition of their craft: it is the science and art of designing and making, with economy and elegance...structures so that they ca safely resist the forces to which they may be subjected."
Mr Vanderburg proposes an adaptation of this definition for software. The juxtaposed ideals of science and art, designing and making, with economy and elegance are preserved, regarding "systems so that they can readily adapt to the situations to which they may be subjected."
From Royce's paper that introduced the waterfall process, Mr. Vanderburg lifts an excellent quote: "The testing phase...is the first event for which timing, storage, input/output, transfer, etc. are experienced as distinguished from analyzed. These phenomena are not precisely analyzable. Yet, if these phenomena fail to satisfy the various external contraints, then invariably a major redesign is required..."
On the analogy between software development and engineering: In structural engineering, engineers produce a design which they provide laborers to implement a physical structure. Applied to software engineering, we have so far been unable to come up with a way to describe a design in such a way that is comprehensible, sufficient, and precise enough for programmers to use without redesigning a lot of it, which they're not trained to do because they're not engineers (in a strict interpretation of the preceding process model. Let the software developers be the engineers and designers; customers don't care about source code. Source code is the design.
He concludes with a link to a paper on Agile methodology, with a simplified diagram of the process. He describes it as "as far on the empirical process design as you can get, but it's no less disciplined, no less rational for being empirical rather than defined."

A Simple Model of Agile Software Processes - Glenn Vanderburg

All in all, it was a very nice presentation, and definitely worth 50 minutes of your time.

Iterating over many-to-many

2010-08-29T00:22:00.002-04:00

I have a django data model that, amongst other things, defines a many-to-many relationship with itself. This post is about solving the problem of iterating the objects related to a particular object in the model.

A self-referential many-to-many relationship is pretty peculiar and deserves some coverage by itself. In order to better grasp what it means, let's examine a concrete example. Consider a social networking site, like Facebook. The relationship between users and their friends is a self-referential, many-to-many relationship. The objects of this relationship are users, and users' friends, the connections between users, represent the relationships that we are interested in.

The connections between users are also significant in their own right. In addition to representing a connection between two users, the relation is also able to keep track of when friends met, how they met, etc.

In the facebook example, the relationship is bi-directional. Facebook friends are always on each others' friends list. This isn't always true. The twitter model, for example, allows one user to follow another without a reverse relationship.

In my case, my self-referential many-to-many relationship has some special properties:

Relationships are not bi-directional
Objects can have no relation to other objects
The relationships have weight
If an object has relationships to one or more other objects, the weights of all those relationships sum to 100%, or 1.
An object could theoretically be a child of itself, directly or indirectly. This would lead to a sort of infinite loop as the relationships are traversed. The sum of all the related objects might be able to be calculated to a specific value of infinite sums are used; however, we are not going to allow an object to be a child of itself.

When iterating over the relationships of a specific object, none of the properties we have listed preclude the possibility that we could visit the same object more than once. One optimization we can apply is to keep track of which objects have been visited so they only have to be traversed once.

I think it can be represented by a weighted, directed tree. The last point in the above bulleted list technically means that the graph contains no cycles. A cycle is basically a loop in the graph, that would occur if an object was allowed to point to itself (directly or indirectly). A graph with no cycles is a special graph that is better known as a tree.

A python instance method is suited to solving this problem, but it's not thread-safe. It is going to rely on a class variable to keep track of previously-visited objects. It could be made to be thread-safe by using objects instantiated from a local copy of the class itself, but as far as I know this possible to explicitly code into the class method; I have to admit that my knowledge of python is limited.

A starter instance method should be used that initializes a hash that keeps track of objects that have already been visited. Then, it calls another recursive instance method that does most of the work. This method visits each related object. First it checks that an object has not already been visited. If it has not, it checks to see if the object has relationships to any other objects. If not, it does some calculations based on the path taken to that object. If it has other relationships, then it does some work to help keep track of the path taken and calls itself on all of the children objects.

Collaborative network shares without compromising security

2010-08-27T17:18:00.000-04:00

I'm in the process of working out a few kinks of setting up Ubuntu services for a small business network. We have successfully used a web server running mediawiki, and some basic NFS services so far. A special type of network share that I need provided by my file server is one in which users are free to read and write each others' files. I'm not sure if NFS is best suited to the task, or if this is something that's more suited to Samba.

The default umask set in /etc/profile of an Ubuntu workstation is 022. This lets users read and write their own files, and allow other users in any group to read their files. This is perfectly fine in most situations; however, we need a shared workspace where users can edit each others' files. I could set the umask to 002, or even 000, but I doubt that's a good practice. This means that on each local system, other users could edit files in each others' home directories. Actually, I'm not even sure if this is true, if the proper permissions are set on users' top-level home directories, but files under those directories would have too-loose permissions set on them.

In general, I want to keep the default umask of 022, but I want a specific share (whether it's NFS, Samba, or something else entirely) to behave like umask 000. I know I could do this in Samba, but my instincts tell me that there is probably a better way.

Could you please share how you provide this service on your networks? What are some good practices to follow? Are there any risks to providing this type of service that aren't immediately apparent?

NFS On Lucid Lynx

2010-08-16T15:38:00.003-04:00

Ubuntu Docs: Setting Up NFS

The above Ubuntu guide is pretty thorough. The following is a summary of the parts that were relative to me.

Firstly, servers must install the package nfs-kernel-server and clients must install nfs-client.

Some best practices to follow are to export nfs shares from /export, and shares in here are mounted bound to other file systems, like so:

# mount --bind /home/users /export/users

or, in fstab:

/media/maxtor/apt-mirror /export/apt-mirror none bind 0 0

Also, a line has to be added to /etc/exports corresponding to the export path, like so:

/export/apt-mirror 10.1.10.0/24(rw,nohide,insecure,no_subtree_check,async)

Restart the nfs service, and the server is all set up. To set up the client, add a line to /etc/fstab like so:

10.1.10.94:/export/apt-mirror /var/spool/apt-mirror/mirror nfs rw 0 0

Learning RT from scratch

2010-07-29T00:10:00.049-04:00

RT is a self-proclaimed hairy beast. The documentation doesn't provide a concise introduction to the concepts, so here is my attempt at providing the most concise, but all-encompassing description of RT. It's pretty directly ripped from the RT wiki, with some fluff cut out.

The wiki home page

it's big, and learning how to set it up can take some time.

please plan to spend some quality time with the UserManual before you try to go live; RT is big and hairy, and you probably can't get your head around all of it in one sitting.

manual introduction

A Ticket is the central object in RT, the thing that needs to get done. Tickets have metadata attached to them such as an owner, status, and queue - we'll get to all that in a minute.

A Queue is the central administrative domain of RT; it keeps things organized. As the name implies, it's a line of tickets waiting to be worked on, but it's also, to some extent, the ticket's category. For instance, you might have the right to create, delete, and comment on tickets in the Foo queue, but only the right to comment on tickets in the Bar queue.

A ticket's history is what it sounds like: everything that's happened to a ticket. Facets of ticket history could include when the ticket was created, how it has changed, and any comments about it or replies to it. Like real history, ticket history cannot be changed (at least not without messing up the space-time continuum), so be aware that any comments you make about a ticket are permanent.

Ticket updates can take one of two forms. A reply is a public remark that a requestor can see. A comment is a private note for staff not visible to the requestor. This is useful when you want to be tactful but still convey important information, like, "This requestor is an investor, so be nice" or "This user's request mentioned his 'PC' but he really has a Mac."

Priority, how important a ticket is, is represented as numerical scale from 0-99, with 99 being the highest priority. ManualAdministration contains a useful model for setting organizational policy for priority.

The status drop-down menu offers several choices for classifying each ticket. Here's what they mean:

NEW: the ticket has just been created and hasn't been touched yet.
OPEN: the ticket is getting worked on
STALLED: due to circumstances beyond your control (waiting for the requestor to respond, waiting for the owner to return from Sri Lanka), the ticket isn't getting worked on right now. It will open again when someone adds a comment or reply.
RESOLVED: hooray! Work on the ticket has been completed and is out of everyone's hair.
REJECTED: the ticket is not the staff's problem and is not going to be resolved, but is, for some reason, worth recording in the system. For instance, if an employee asks approval for something ridiculous, you can reject the ticket, but it will stay in the database as evidence that the employee makes silly requests.
DELETED: the ticket never should have been in the system (maybe it was spam, a list of passwords, etc) -- and is now being zapped for good.

A watcher is someone who is interested in a ticket. You'll find these filed under "People" when you're looking at a ticket. A watcher of a ticket normally gets email-copies of all the replies to the ticket. The watchers who are staff-members get copies of the comments too. There are several types, or roles, of watchers:

OWNER: the person responsible for the ticket and its resolution. Each ticket can have only one owner. As a ticket it worked on, it may be passed from one owner to another.
REQUESTOR: the person or people who asked for something to get done; the ticket's reason for being. The Requestor normally gets copies of all replies to the ticket, but no comments.
CC: someone (or someones) who should get copies of any replies that go to the requestor. This might be the requestor's boss, sales rep, etc. This person will see the emails but doesn't necessarily have the right to work on the ticket.
ADMINCC: a Cc or Ccs that also gets copies of comments and generally are allowed to work on the ticket.

You can also, as you work on a ticket, define its relationship to something. Relationships in RT can be ticket-to-ticket, but can also link tickets to external items like URLs or FedEx shipping numbers. For the sake of simplicity, we'll refer only to tickets in the following explanation of relationship types:

DEPENDS ON: the ticket can't be resolved unless another ticket is also resolved. The converse is depended on by.
REFERS TO: the ticket doesn't need the other ticket, but it would sure be useful for you to look at it. The converse is referred to by.
PARENT: a big, general ticket ("Move house.")
CHILD: a subproject of a parent ("Hire movers." "Pack." "Eat pizza.")

CustomFields are, yes, database fields that your organization can make up according to its needs. For details, see the Custom Fields section in ManualAdministration. An example custom field might be Resolution: FIXED, WORKSFORME, WONTFIX, UPSTREAM, etc, etc... (If you think that example was stolen from Bugzilla, you have much to go on :-)

Scrips are custom notifications that will automatically take Action X in response to Condition Y. You could have RT notify the Requestor when a ticket is resolved, for instance, or have RT page you when your boss submits a request. RT comes with many standard scrips, and you can define your own. For details, see the Scrips section in ManualAdministration. And no, we didn't misspell "scripts." We made up a new word. Try it, it's fun.

Book review: Beginning Ubuntu Linux, Fourth Edition

2010-06-15T23:16:00.001-04:00

This is a book review of Beginning Ubuntu Linux: From Novice to Professional, Fourth Edition. The print ISBN is 978-1-4302-1999-6 and the web ISBN is 1-4302-1999-8.

I read the contents of this book almost cover to cover; I only skipped the multimedia section. I read this because I was interested in reading Automating Linux and Unix System Administration, which apparently is last in a sequence that starts with this book, Beginning Ubuntu Linux.

In short, Beginning Ubuntu is an above average introduction to the world of Linux. Anybody who's a windows power user can probably poke around and figure out most things on there own, but parts 4 and 7, The Shell and Beyond and Keeping Your System Running, should be mandatory reading for new Linux users. Part 3, The No Nonsense Starting Guide has some parts which only apply in specific situation, but it's worth reading once.

Part 4, The Shell and Beyond, is a concise and clear introduction to the BASH shell, the Filesystem Hierarchy Standard, and in general, the UNIX way of doing things.

It's not the kind of book I would want to keep around as a reference, at least not for very long. It's not thorough enough on any one subject to make it a good reference. The tone is easy and almost conversational. If you are savvy enough to keep a Windows installation running virus free, and perhaps if you feel comfortable re-installing windows from scratch, then you could probably digest this book pretty easily.

There are two things I think are missing from this book. One is a procedure for surviving a major upgrade. Every ubuntu user is going to do this sooner or later. A novice could use some advice interpreting the various dialogs that will pop up during a distribution upgrade.

The second missing piece is a guide about compiling programs from source. Even if it didn't get a full chapter, novices deserve an overview of how programs are built.

Sneakernet Ubuntu Server Updates

2010-06-07T08:31:00.010-04:00

After initially deploying an ubuntu server high side of an air gap (wikipedia), there will come a time when packages need to be added and/or updated. This article attempts to outline a process to build and keep up-to-date a private local repository.

In short, I will configure the packages on an internet-facing staging server, and copy the downloaded packages to the private server. The packages themselves contain enough data to build the private repository. I've adapted the procedure from Odzangba's How To. The main requirement for this to work is that you have access to a staging server on the low side, which will maintain an identical package configuration. This requirement wasn't bad for me, because a staging server is useful for a million other reasons. I run it in a virtual machine on my internet-facing development workstation. Either the low and high servers have to start in an identical configuration, or you have to be assured that /var/cache/apt/archives hasn't been cleared since they may have diverged. For the following procedure we'll assume the two servers are identical. You also need some medium of transferring data between the two hosts. This is where the term "sneakernet" originated. I'm lucky that I don't have to run floppies between two server rooms; thanks to the wonders of modern technology, all i have to do is shift a thumb drive from one USB hub to another. If you are going to be doing this frequently, I would suggest making the sneakernet step as quick and easy as possible.

Lastly, you will also need to get build-essentials. It is required to have this on the private server before you get started.

Just so there is no ambiguity, I'll refer to the private host as high and the internet-facing host as low. lowadmin is the user on the low side, and highadmin is the user on the high side. /home/highadmin/localrepo is the path to the repository on the high side, and this must be added to /etc/apt/sources.list. The only line the the sources.list file on my high server is

deb file:/home/highadmin/localrepo/ /

OK! So, assuming low and high have matching package configurations, this is how to add and/or update packages to the low side and sync those changes to the high side.

The low and high servers are identical. I want to add and/or update packages, so I'll execute the following commands on the low host...
sudo apt-get update
sudo apt-get upgrade
and maybe...
sudo apt-get install foobar spam
Again, on the low host...
rsync /var/cache/apt/archives/*.deb /media/thumbdrive
I also like to add -vh (verbose, human-readable). Maybe rsync isn't necessary here, but I'm just in the habit of using it all the time. Also, my thumbdrive isn't usually passed directly to the virtual staging server, so my command looks more like this...
rsync -vh /var/cache/apt/archives/*.deb lowadmin@devbox:/media/thumbdrive
Sneakernet step. I move the thumbdrive two inches into another hub, and mount it on the high side server, then...
rsync -vh /media/thumbdrive/*.deb /home/highadmin/localrepo/
sudo dpkg-scanpackages /home/highadmin/localrepo/ /dev/null | gzip -9c > Packages.gz
sudo apt-get update
sudo apt-get upgrade
This is where the magic happens. The second command uses dpkg-scanpackges to create the crucial Packages file. For further reference, see the man page.

That's it; the high side is now up-to-date.

Ubuntu, vim and python

2010-03-27T04:24:00.001-04:00

Until now, I had never used vi or vim as anything more than an emergency configuration file editor. The following two posts helped me leverage some of the more advanced features and plugins of vim to make it a nice python, fully-featured python editor. All credit goes to the following two posts. Most of the content below is copy-pasted directly from them. It's not my intention to plagiarize; this is just a spot for me to take notes, so that I could repeat the vim configuration process as painlessly as possible.

http://dancingpenguinsoflight.com/2009/02/python-and-vim-make-your-own-ide/
http://henry.precheur.org/2008/4/18/Indenting_Python_with_VIM.html

Create the directory ~/.vim/ftplugin; create a file named ~/.vim/ftplugin/python.vim containing:

setlocal tabstop=4
setlocal softtabstop=4
setlocal shiftwidth=4
setlocal textwidth=80
setlocal smarttab
setlocal expandtab

To get this working, simply add:

syntax on
set number
filetype plugin indent on

to your ~/.vimrc file

Additionally, you need to download the latest version of this script, placing it in ~/.vim/indent/python.vim. This will make indentation much less of a chore, and more of a pleasure. Be sure to also check out this blog post, wherein Henry Prêcheur explains how to improve indentation for comments as well.

To improve the formatting and display of improper indentation, as well as special highlighting for things like string formatting, download the latest version of this script, and place it in ~/.vim/syntax/python.vim.

Philosophy of Managing Programmers

2009-12-13T16:11:00.001-05:00

http://it.toolbox.com/blogs/irm-blog/theory-p-the-philosophy-of-managing-programmers-4993

This article gave me some insight in to my own behavior.

Some of the commenters on his site have pointed out that he portrays programmers in a negative light; I disagree and think that this article contains advice that would effectively help protect a business from making the wrong decision about projects.

It also contains advice for programmers explaining how to avoid making some bad business mistakes.

Search your command history

2009-12-13T15:23:00.003-05:00

history is the command to "utilize information from previous lines" (from the man page). My most common usage is like this:

history | grep -i pattern

Maybe now that I've recorded it here, I will remember it, and won't have to reference it for the millionth time.

Finding the strongest security and best value in remote storage

2009-11-19T23:21:00.000-05:00

Data can be priceless. No expense should be spared to prioritize the following two objectives:

Information privacy
Information safety and redundancy

I'm trying to come up with a strategy that addresses these needs, and I'd like a little feedback. Also, corrections are much appreciated and I'll do my best to keep this up to date based on any information in the comments.

Data needs to be private. Businesses count on trade secrets, and keeping information private is what keeps a business running. Eventually, sensitive information will end up on a host that is in contact with the internet. The contact could be indirect, temporal, or both. Your privacy should be hedged on the assumption that any host is in danger of being compromised if it's ever in contact with an internet-facing network. The only way to design a system with the highest-possible degree of security is to operate with this assumption in mind. Information also needs to be safe. Nothing less than nuclear holocaust should lead to information loss.

With the right application of freely available software, this is not only possible, but cheap.

The most important program needed is the GNU Privacy Guard, or, best known by it's command-line name, gpg. gpg is a free implementation of the OpenPGP standard. When it comes to keeping your information private, OpenPGP is the de facto standard on the cutting edge of computer science. As far as academia is concerned, even theoretical liabilities have all been weeded out since 1996. This encryption is invulnerable, except possibly to the military, or corporate black ops. The proof is statistical; if you applied every supercomputer in the world to the task of analyzing a data with this encryption, it would take lifetimes to find the decode. OpenPGP also provides a second layer of privacy; it allows you to sign your data so that it is protected from alteration in transit and in storage. With OpenPGP you are assured that data you store is meaningless to anyone without the key. You are assured that the data you retrieve is identical to the data in your original source (Schneier, Bruce (1995-10-09). Applied Cryptography. New York : Wiley . p. 587. ISBN 0471117099 .).

The benefit of secure encryption, from the perspective of remote backup, is that you do not need to trust ANY hosts outside of your network with your information. If your storage provider is in the news because the security was cracked by hackers, you know your data will stay private because you've already taken precautions against the host becoming compromised.

Encryption is still very much relevant to locally stored, physical media. Locks can be picked. Cars can be stolen. Packages can get lost. Every piece of media that is not bolted down, and even some that are, could be the source of a potential leak.

The next topic is a discussion of value. The industry benchmark when it comes to the cost of storage is Amazon Simple Storage Service (S3). To back up a total of 1TB of data, and transfer in 100GB of data per month results in a cost of $1920 per year. S3 features a pay-what-you-use model, which means your needs are guaranteed to be met as they grow. A cheaper option is to use a less permissive backup solution like Backblaze. With Backblaze, you are purchasing an unlimited backup license for one computer for $50 per year. This computer must be running a Microsoft Windows operating system, so there is a one-time investment in a Windows license, and there must be a Windows host on your network. The disadvantage of requiring a Windows computer on your network is small compared to the license terms: if the computer you license to Backblaze provides services in your intranet, your complacency with Backblaze's terms is questionable, and your service could be terminated depending on the interpretation of their license agreement.

So, the cost range of acquiring third-party remote storage services has a wide range of $50 to about $2000 per year. The cost of implementing the same solution in house is somewhere in between these two numbers, but it's harder to pin down because it involves man-hours designing, building and maintaining storage systems with a wide range of strict requirements for data integrity.

When using third-party remote storage, you're basically covered if you subscribe to two distinct services; if one suffers a catastrophic failure, your data is still hosted on another site, and hopefully locally as well. When implementing backup first-party, it's your responsibility to maintain data integrity as well as the physical integrity of the system. Parts and computers must be maintained, software must be updated, and the whole system must be protected from physical harm, such as a fire.

Let's estimate the cost in hardware to backup, redundantly, 1TB of data to be $800. At a rate of $18/hour, it only takes 60 hours over the course of a year to match the cost of Amazon S3 services, and an additional 100 man hours to match the cost of having two distinctly located buckets with S3. Consider that in order to approach the data safety offered by a service like S3, you would have maintain distinctly located data repositories. This involves maintaining remote hosts and connections between the hosts. Alternatively, data could be physically moved on a regular schedule, but this puts an increased strain on the hardware and results in a higher hardware cost to protect the data from physical shock, purchase special cases, or increase hardware redundancy. From this perspective, S3 seems like a bargain. Costs of S3 are also likely to fall in proportion to the decreased cost of storage, but the cost of man hours puts a price floor on the service; in other words, don't expect prices of S3 storage to fall below $1000 per TB in the next three to five years.

For any organization that depends on data, information privacy and safety are crucial. Here is everything I need to know (I think) to make sure that these issues are addressed and policies are in place to meet these needs for a business.

First apps and models are validated!

2009-10-16T17:19:00.001-04:00

My very first apps and models have been validated. I could have saved a lot of frustration had I only read through the entire tutorial. So, the moral of the story is, RTFM.

Specifically, I needed to use functions that are included as part of manage.py, and not try to make any glue myself.

I will add to this post, on Monday, with details on how exactly I got my first Django project started.

Installing python packages

2009-10-15T17:11:00.001-04:00

There is a vast library of python code available to tap into. It's all available at the Python Package Index (pypi).

In order to connect the Django app to the PostgreSQL database, I had to install psycopg2, which is Python's PostgreSQL database adapter.

Debian makes this easy for us because python-psycopg2 is maintained as part of Debian's repos. If this was not the case, then I would have to wget the tarball, extract it, and run 'python setup.py install' for the package. In order for that command to work, one has to have the dev package for your version of python installed. In my case, that would be 'python2.5-dev.' This package provides header files for building your own modules from scratch.

There's a lot of jargon in this post which would be familiar to any intermediate Linux user. I will come back at a later date and provide a glossary similar to the bullet points I added to my git post.

Version control

2009-10-15T16:31:00.000-04:00

I will be tracking this project with the git version control system.

Revision Control, as it is sometimes called, allows you to commit a group of documents to a repository, in order to keep track of every revision.

repository - the server/location that collects and keeps track of your project
commit - to push a snapshot of the current state of your project
revision - a serial index that tracks every commit you make to the repository

Revision Control has many benefits. The first is that you store every revision of your project, so that you can roll back to any revision number. The next benefit is the ability to branch your code. Your mainline code is typically called the trunk. If you decide to change some things around, or start writing a new module to your project, you can create a branch which can be tracked separately, and then merged with the trunk. Git allows you to diff any two versions of a file, which highlights all of the differences between the two files in a way that's easy to browse and understand. Many users can clone your repository, work on the same files simultaneously, and git provides you with a very nice set of tools to merge all the changes in a way that makes sense. The last benefit to using git is that it's a distributed system: every clone is a complete copy of the entire repository, which is securely copied (i.e. the data is verified as it's copied). This makes git clone an excellent tool to back up a project.

I will be using git to keep track of different versions, to document my work, and to back it up.

Django - Design Your Model

2009-10-15T11:56:00.000-04:00

The first step that I'm going to take to designing my web app is to start building up the model. Reference for this section starts with the Design Your Model section of the Django overview, and also the related section Data-Model Syntax.

The model is one of three parts that build up the Model-View-Controller architectural pattern. Basically, the model is a description of your data. Let's say you're designing a social networking application. In plain English, your model would start off something like, "The basic unit of data that we are working with is a member of the site. A member has a first name, last name, password and chosen user name. Members are related to other members in a many-to-many relationship. Members can be friends with other members, they can ignore other members, etc. By default, a member's list of friends is sorted by the order in which they were added..." and your model would continue. The formal model is nothing more than a translation of this natural language model into python.

The view component of an MVC web app tells the browser how pages are displayed. Mostly, the view is made up of templates that have components of the model plugged in.

The controller of an MVC web app handles page requests and decides what to do with components of the model. For example, if you went to visit the home page of our social networking example, it would check to see if you had an existing session, and if not, it will grab a log in page provided by the view. After a user enters this info, the controller takes care of several steps:

It attempts to authenticate the user against existing users in the database
If authentication is successful, it grabs the appropriate components of the model
It plugs those components into a template provided by the view
It sends the template, complete with info composed from the database, to the user's browser.

Armed with this information, I am going to begin writing parts of the model, and then write some python scripts to import my existing data into my Django web app.

For further reference, see Interacting with a Database: Models from Definitive Guide to Django; however, please note that, although this text is less than two years old, different versions of Django can change dramatically. The most up-to-date reference is always available at http://www.djangoproject.com/.

Server setup

2009-10-14T11:37:00.000-04:00

I'm going to attempt to chronicle the events of me learning how to create web apps. As of now, I'm responsible for creating everything from scratch. At this point, I've already sketched a very rough outline of how my first web app is going to turn out; I have chosen to build it on the Django framework, with a PostgreSQL database, running on a Debian server. All of those are new to me. Well, I have had some experience running Ubuntu on my home desktop, and as a simple SSH/file server, so Debian isn't entirely new to me.

So far, I have installed Debian with kernel 2.6.26-2-amd64, and all of the software I need. Hopefully, by today, I should have some equivalent of "Hello, world" that I can access through a browser from some other computer on the LAN, and maybe start populating the database with some sample data.

I'm skimming Dive Into Python, The Definitive Guide to Django, and the PostgreSQL docs as I go along.