Saturday, March 28, 2009

365 DoC - W1, D1 - Subset Permutations

Problem: based on a given finite set of elements, find every possible combination of X elements, where X indicates 1 to X.

Example: in the finite set of elements (A, B, C), all possible combinations of up to three elements would be ((A), (B), (C), (A, B), (A, C), (B, C), (A, B, C)) - this assumes that we would treat (A, B) and (B, A) as the same permutation of a 2-element subset, etc.

Solution: if you think of the set of elements as digits in a number system, then you simply need to enumerate between 'zero' and the max for the number of elements in each permutation, while eliminating duplicates. For example, decimal (base-10) is a number system comprised of a set of 10 possible elements: the digits 0 through 9. So, given this set, if we wanted to find all the permutations of up to 3 elements, we simply count from 0 to 999, and eliminate duplicates/repeats, ie: '999' would be eliminated because we would already have '9' in the list of possible permutations... likewise, '21' would be omitted, since at that point we'd already have '12' in the set of possible permutations of digit combinations ('21' is just a re-arrangement of the digits in '12').

Sounds complicated, but we really just need two things: a function to 'increment' a set of elements (the permutation), and an easy/efficient way to check for duplicates/repetitions.

The first item will end up being a simple implementation of an elementary school place-value lesson... the pseudo-code will look something like this:

possible elements: A, B, C
permutation to 'increment': (A, C)

- set the current 'place' we're working on to zero (the first 'digit', 'C' in this case if we're working right-to-left)
- is the 'value' of the element at the current place the last possible value in the set of possible elements?
yes: set the value of the current place to the first element in the set, move the place up one, and repeat
no: set the value of the current place to the next element in the set

'A, C' would be "incremented" to 'B, A'

This is a actually a good candidate for recursion, although I didn't really plan on talking about recursion in today's excercise... here's a simple implementation of the above 'algorithm' in PHP...


$possibleElements = array('A', 'B', 'C');

function incrementPermutation(&$perm, $place = 0)
{
global $possibleElements;

if ( !isset($perm[$place]) )
$perm[$place] = $possibleElements[0];
else {
$valueIndex = array_search($perm[$place], $possibleElements) + 1;

if ( $valueIndex < count($possibleElements) )
$perm[$place] = $possibleElements[$valueIndex];
else {
$perm[$place] = $possibleElements[0];

$place ++;

incrementPermutation($perm, $place);
}
}
}


The implementation is slightly incomplete - there's not argument checking (the $perms parameter needs to be an array), no checking of the $possibleElements array, and there's some other updates we could make so it's more general-purpose, but it will work as-is. PHP might optimize it out anyways, but for the sake of writing good code we've kept the $possibleElements declaration/initialization outside of the function body so that it's not re-declared every time we (potentially) recurse.

Instead of commenting the function to explain it's operation, let's dissect it piece by piece:

$possibleElements = array('A', 'B', 'C');

This is just setting up an array to hold the full set of elements from which we'll be generating permutations - if we were working with a real numbering system, this would hold our digits, in order... ie: array('0', '1', ..., '9');

function incrementPermutation(&$perm, $place = 0)
{
global $possibleElements;

Our function declaration... we're passing $perm, which is an array representing the existing permutation to increment, by reference so we don't need to write clumsy $var = func($var) type statements all over. $place defaults to zero so that outside of the recursive calls, ie: elsewhere in our code, we just call incrementPermutation($perm) without worrying about explicitly telling the function to start at the "least-significant" spot in the permutation.

Again using the example of a numbering system, $place indicates the place-position in the number that we're working with. This corresponds to the 'ones' or 'tens' or 'hundreds' position in a decimal numbering system, if you remember back to the explanation you got in grade school.

if ( !isset($perm[$place]) )
$perm[$place] = $possibleElements[0];

First we check if the $place we're working with has already been set/defined... if not, incrementing is easy: we just set it to the first element in $possibleElements and we're done.

else {
$valueIndex = array_search($perm[$place], $possibleElements) + 1;

Otherwise, we'll need to do some real work - first we need to figure out where in the set of possible elements the current value of the element at $place in our permutation resides. We increment that because in both places where we're about to use it we need to increment it anyways, so we might as well do that right off the bat...

if ( $valueIndex < count($possibleElements) )
$perm[$place] = $possibleElements[$valueIndex];

This expression, if True, indicates that we don't need to "carry-over" anything - we can increment the value at the current place without running out of elements in the original set to use. $valueIndex represents the next possible index/key in the array of possible elements. If it's not less than the size of $possibleElements, that means we're already at the last possible element and need to 'carry-over'. Otherwise, it will 'point' to the next possible element, which is used as the incremented value for this $place in our permutation.

else {
$perm[$place] = $possibleElements[0];

$place ++;

incrementPermutation($perm, $place);
}

If we do have to carry-over a value, we set the current place's value to the first in the set of possible elements, increment the $place variable, and recurse, passing in the new value of $place, in order to perform the whole increment operation on the next place over.

Next, we need to deal with the task of identifying duplicates/repetitions in our permutations. The solution is relatively simple... we're going to write a function that generates a numerical value for our resulting permutation, based on a bitmask that assigns a different (bit-)value to each possible element in our original set. To generate the value of our permutation, we perform a bitwise OR on the bits corresponding to each 'digit' in the permutation, and then obtain the resulting decimal representation of the number. If this matches the value of a permutation already in our 'found permutations' array, which we're tracking, then we can safely ignore it.

Here's the function:

function permutationValue($perm)
{
global $possibleElements;

$value = 0;

foreach ( $perm as $element )
$value |= 1 << array_search($element, $possibleElements);

return $value;
}

It's pretty simple - we're just finding the position (index) of each element in our permutation, and using that as a bitshift operand in our |= operation, which 'adds' bits to $value.

And that's pretty much it - using these two functions, we can enumerate all possible combinations of the elements in a given, arbitrary set, and by keeping track of our permutation 'values', we can check for duplicates/repetition before adding each found permutation to the final list (array) of permutations discovered.

Here's the complete version of the code, with a built-in example... you can see the output here.

// define the members of the entire set
$possibleElements = array('A', 'B', 'C');

// the max size of a permutation is the number of elements in the original set
$maxSetSize = count($possibleElements);

// initialize an array to store the found permutations
$permutations = array();

// initialize an array to store the "values" of each permutation found
$permutationValues = array();

// determine the "value" of a permutation
function permutationValue($perm)
{
global $possibleElements;

$value = 0;

foreach ( $perm as $element )
$value |= (1 << array_search($element, $possibleElements));

return $value;
}

// "increment" a permutation, using each element of the original/whole set as a possible "digit"
// in our arbitrary number system
function incrementPermutation(&$perm, $place = 0)
{
global $possibleElements;

if ( !isset($perm[$place]) )
$perm[$place] = $possibleElements[0];
else {
$valueIndex = array_search($perm[$place], $possibleElements) + 1;

if ( $valueIndex < count($possibleElements) )
$perm[$place] = $possibleElements[$valueIndex];
else {
$perm[$place] = $possibleElements[0];

$place ++;

incrementPermutation($perm, $place);
}
}
}

$currentPerm = array();

while ( count($currentPerm) <= $maxSetSize ) {
// "increment" the current permutation
incrementPermutation($currentPerm);

// calculate the value of the new permutation
$currentPermValue = permutationValue($currentPerm);

// if it's not already in our array of values, add the new
// permutation to the list of ones we've found
if ( FALSE === array_search($currentPermValue, $permutationValues)) {
array_push($permutations, $currentPerm);
array_push($permutationValues, $currentPermValue);
}
}

// Dump out the final list of permutations in semi-pretty/human-readable format
print_r($permutations);

365 Days of Code

That's right... one piece of working, usable, non-trivial code (ie: it actually does something useful), every single day, for an entire year.

Ambitious? Maybe. Necessary? I think so... you see, at some unknown point in recent history, I made a conscious decision to strive to be The Best at what I do. And what I do, is Code. It's my Art. I don't do this for a living because it pays big bucks or I'm trying to become a millionaire, I don't do it because I figured computers would be a safe bet for finding employment, or anything silly like that - I have another ridiculous explanation: for some reason, I love programming. Yes, I'm a big geek. And because it's something I love, my goal is simpler, although much harder to achieve: perfection, or as close as I can get, in the Art of Code.

Impossible? Definitely... but that's besides the point, which is that no one ever became a rockstar by sitting on their ass and dreaming about how awesome it would be - they did it by playing their guitar for hours on end every single bloody day until their fingers bled and they were really, really, ridiculously good at it. And then (hopefully), they practice some more, because even though everyone else already knows they're awesome, they still don't think it's good enough.

And so, we come to the real point: I'm not going to become the next Torvalds or Kernighan or Knuth (yes, these are my heroes, that's how much of a nerd I am - there, I said it) by working a lot of overtime, reading, or cranking out a lot of code, but only by making a conscious effort to hone my skills, any and every relevant way possible, every single day. No samurai was ever born on a couch.

I need to turn my Art (which, althogh already pretty awesome, is still just stick people playing kick-ball on a piece of loose-leaf lined paper), into something closer to Code Budo. The Way of the Code Warrior. The Art of the Software Sandbenders.

So. One (at least) piece of working code, every single day, for an entire year.

And here... we... go :)

Thursday, May 15, 2008

The Interruptable Sequence Pattern

The interruptable sequence is a web programming design pattern I recently identified and used in a site I'm developing. I tried to locate existing descriptions/definitions of this pattern with no success - I looked for 'interruptable sequence', 'tutorial pattern', 'sequence pattern', etc. without finding anything similar to what I'm about to describe, so I'm posting about it here in the hope that someone else will find it useful.

Software design patterns are an idea that's been around for quite a while... there's plenty of literature on the topic, not the least of which is 'Design Patterns: Elements of Reusable Object-Oriented Software' by the 'Gang of Four' (ISBN: 0201633612, ISBN-13: 978-0201633610). However, web-patterns are a more recent topic, and although there's a little bit of content spread around the internet, there doesn't seem to be an authoritative or even central/complete catalog. As such, I'm going to try and collect what I can here, especially since I've been very much into the software design and modeling area of study lately.

So, without further ado, the Interruptable Sequence pattern. (I've attempted to follow the same format as the pattern catalog contained in Design Patterns, so hopefully this ends up being relatively coherent and/or of a familiar format to some of you...)

Note: In the context of this (and future) articles, a 'Web Pattern' is a software pattern that applies specifically to software whose environment is the web, ie: sites or webapps...

Interruptable Sequence

Classification
A behavioral pattern with a cross-request scope.

Intent
Supports a sequence of actions as a progression of separate page requests, which can be started, stopped, resumed, exited and completed at any point throughout a user's session.

Also Known As
Tutorial Decoration

Motivation

Let's assume that you have created a site that requires a tutorial or walkthrough, to guide new users through using the basic functionality. You want the user to be able to 'follow along' with your instructions, but you also don't want them locked in to the tutorial until it's complete - they should be able to exit and resume the tutorial at will, etc.

There are some requirements for such a sequence that immediately come to mind:

  • - we want the user to be able to enter and exit the sequence at will - therefore upon exit the state of the sequence should be preserved, so that on re-entry, the user is directed to the same point in the sequence where they left off.
  • - we want to be able to simply 'decorate' existing functionality with messages related to the sequence - we don't want to duplicate pages/fragments for the sake of presenting them within the sequence.
  • - the logic for a given page should know that the user is participating in a sequence (or not), and render the page (or rather, the sequence-related elements) appropriately.
  • - ideally, the only customization of the sequence implementation that we would need to do each time we want to setup a sequence is to define each step in the sequence, ie: via a small set of information required for each step, such as the page uri, the sequence message, the sequence this step belongs to, and the index of the step if we're not simply appending steps in their 'native' order.
The pattern solves the problem as follows...
  • - The code library for the pattern defines two main functions: sequence_manage, which is run at the beginning of every user-facing page request and sequence, which is run during every request that is part of the sequence.
  • - sequence_manage is responsible for initializing the sequence-related data structure within the session store, and more importantly, for storing certain variables related to the state of the sequence.
  • - sequence is responsible for realizing each successive step of the sequence.
  • - The pattern's code library will also define an auxillary function: sequence_append_step, used to setup the sequence itself.
Applicability
Use the Interruptable Sequence when:
  • - you want to define a sequence of page requests that can have auxillary information displayed with each step, and the sequence follows some kind of progression, ie: a tutorial.
  • - you want the user to be able to leave and re-enter the sequence via normal navigation elements at any point.
  • - you want the state of the sequence to be preserved between separate periods of the user being 'inside' the sequence
Structure
Note: diagram isn't up to snuff, this was done in a rush via Gliffy, but I'll update and post a better version eventually...





Participants
  • - a session store - used to persist sequence state information across requests.
  • - sequence_append_step, sequence_manage and sequence - used by the request handler at various points to execute the necessary sequence logic.
A session store is assumed to be provided already by the existing site or framework code. The other participants are provided by the implementation of the pattern... in the example below, the implementation is targeted at Ruby on Rails, and the pattern is implemented as a collection of methods in a module that is mixed in to a controller class.

Collaborations
A session store is used to store the necessary state information used by sequence_manage and sequence. sequence uses the sequence-step definitions created by sequence_append_step.

Consequences
The main consequence of this pattern is a decoupling of the components of the sequence (ie: text, sequence-specific navigation elements, logic) from the existing site components that comprise the steps of the sequence (ie: pages of a site or application). This decoupling allows us to independently vary both the sequence itself and the components that comprise the steps without affecting the other. It also allows us to re-use the components used for each step in more than one distinct sequence. Lastly, following the "don't repeat yourself" software engineering philosophy adhered to by any good object-oriented system, it removes the need for any duplicate logic or code/markup that is used to implement the sequence. We are left with a single library of functions/code that allows us to create a sequence, and the only per-sequence coding that needs to be done is to define the steps of the sequence itself and the content of each sequence step (ie: the text that decorates an existing component when displayed as part of a sequence).

Implementation
To implement the pattern, there are a few pieces of data that we need to store in a session:
  • - the current step of the sequence (ie: sequence index)
  • - the name of the sequence that's currently being followed
  • - whether or not the sequence is currently active
  • - the uri that acts as a referrer for the sequence - this is the last uri that was visited by a user before (re-)entering the sequence
  • - the current uri, to be accessed on the next request as the last uri (Note: this data is almost always available via the server-side environment as the HTTP_REFERRER. However, to remove such a dependency, we include code to explicitly store this value in the sample implementation below, instead of relying on the environment of the application/site to provide it implicitly).
Next, we need to write a few main functions to implement the sequence pattern:
  1. A sequence_append_step function that is used by the site/application to define the steps of the sequence. The sequence is treated as a stack of items - each item corresponds to a step in the sequence, and the first item in the sequence 'stack' corresponds to the first step in the sequence. This function will need to take a few arguments: the name of the sequence we're appending this step to, the uri of the step that's being appended, and any other data needed to implement the step... in the example below under Sample Code, the last argument is the name of a template that's rendered as the decoration for this sequence step. The method, timing and location used for storing the data for each step is irrelevant to the pattern itself; as such, either persisting this data between requests somehow (session, rdbms, etc) or calling the step-defining code on every request is a valid answer to this question, the storage method most appropriate will depend on the programmer's specific requirements for speed or storage optimization, etc.
  2. A sequence_manage function that is called at or near the beginning of every request. It's main responsibility is to determine whether we are currently following a sequence or not. It does this by checking if either the previous uri (ie: the referrer, the uri that redirected us here) or the current uri corresponds to a sequence uri. If not, it stores the current uri as the sequence referrer uri, and explicitly disables the sequence (via a sequence_active flag stored in the session), since we're not currently following a sequence. The sequence referrer is used to return the user to the place from which they entered the sequence, once the sequence has been completed. Additionally, this function is responsible for storing the current uri in the session, to be accessed in in the next call to sequence_manage as the last_uri mentioned above.
  3. A sequence function - this is generally activated by visiting a sequence-specific uri, ie: by clicking 'next step' in a sequence or a navigation link that targets (the beginning of) a sequence. The sequence function needs to be passed the sequence step to 'execute', as well as the name of the sequence we're executing, if not already known (ie: when starting a 'new' sequence). This function is responsible for the bulk of the sequence logic... 1) if the named sequence doesn't exist, or if the sequence step being executed corresponds to 'finish', then finish the sequence (reset the sequence step to zero, disable the sequence, and return the user to the sequence_referrer uri), 2) otherwise, explicitly enable the sequence (it may not already be active), set the current sequence step if a step value is passed and redirect the user to the uri corresponding to the (first) current step in the tutorial.
Sample Code

The following is a bare-bones implementation of the pattern written in Ruby, for the Rails web application framework.

We already have session and redirect support, which are the main requirements/dependencies for this implementation of the pattern.

First, the pattern 'library':

module InterruptableSequence

def sequence_append_step(sequence_name, sequence_uri, sequence_data)

# initialize our local sequence data structure(s), if necessary...

if @sequences.nil?
@sequences = {}
end

if @sequences[sequence_name].nil?
@sequences[sequence_name] = {}

@sequences[sequence_name]['uris'] = []
@sequences[sequence_name]['data'] = []
end

# append the step to the sequence...

@sequences[sequence_name]['uris'].push sequence_uri
@sequences[sequence_name]['data'].push sequence_data
end


def sequence_manage

# initialize the session-based sequence data if necessary, as a hash
# so that the only pollution of the session namespace we're doing is adding
# one new name ('sequences')

if session['sequences'].nil?
session['sequences'] = {}
end

# check if we need to store the referrer and disable the sequence

# note: this implementation assumes that the controller action for a
# sequence step is 'sequence', which is how we identify if this request
# is for a sequence page or not

if session['sequences']['last_uri'] \
and session['sequences']['last_uri'].match('^/[^/]+/sequence($|\?)').nil? \
and request.request_uri.match('^/[^/]+/sequence($|\?)').nil?

# this is currently NOT a sequence-step request... store the sequence referrer

session['sequences']['sequence_referrer'] = request.request_uri

# turn off the sequence

session['sequences']['active'] = false
end

# store the request uri, to be used during the next request as the referrer

session['sequences']['last_uri'] = request.request_uri
end


def sequence

# read arguments from request parameters, if present

if ! params[:step].nil?
session['sequences']['step'] = params[:step]
elseif session['sequences']['step'].nil?
session['sequences']['step'] = 0
end

if ! params[:name].nil?
session['sequences']['name'] = params[:name]
end

# check if we're 'finishing' the sequence

if -1 == session['sequences']['step'] \
or @sequences[session['sequences']['name']].nil?
session['sequences']['step'] = 0
session['sequences']['active'] = false

redirect_to session['sequences']['sequence_referrer']
else
session['sequences']['active'] = true

redirect_to @sequences[session['sequences']['name']]['uris'][session['sequences']['step']]
end
end

end

Next, we make use of the pattern library in our main
ApplicationController class:

class ApplicationController < ActionController::Base
include InterruptableSequence

...

before_filter :sequence_manage

...

def initialize
sequence_append_step('my tutorial', '/orders/create', 'my_tutorial_step_one')
sequence_append_step('my tutorial', '/orders/edit', 'my_tutorial_step_two')
end
end

In this example implementation,
sequence_manage is being called on every request via the installed before_filter in the ApplicationController class, from which all other controllers (request handler classes) inherit. The sequence is defined in the constructor of the ApplicationController class, so it will be available in all controllers. sequence_append_step, used to define the sequence, stores all sequence-specific data in @sequences, which is an instance variable of whatever controller object the request is being handled by. Since we're including the InterruptableSequence module in the main ApplicationController class, any controller can call sequence (ie: by visiting /controller_name/sequence) to start/resume a sequence.

The page rendering code can examine session['sequences']['active'] to check if a sequence is currently active... if so, it can use session['sequences']['step'] to pull the sequence-step data value from @sequences, which in this case is the name of a template to render within the page that's being used as part of a sequence.

The same code also uses the step index to construct sequence-specific navigation elements such as 'next' or 'previous'... if we're on the last step, a 'next' link would pass '-1' as the step index to the next request, which would signal the sequence logic to 'finish' the sequence and redirect the user back to where they were when they started the sequence.

Note that the above implementation has been simplified for demonstration purposes - it's lacking some important error checking, etc. Also, certain functionality which is supported by the pattern may not be illustrated above, such as multiple sequence usage, etc.

Known Uses
As already discussed, a website usage tutorial is a good example of a candidate for using the Interruptable Sequence. Another example of an application for this pattern would be an online shopping cart checkout process, whose steps are comprised of cart functionality that already stands on it's own, ie: using the independent 'update shipping address' page within the checkout flow. Alternately, this pattern could be applied to a situation where only part of the sequence is made up of pre-existing functionality, with the balance being created specifically for the sequence (ie: a variation of the online shopping cart example above, where portions of the checkout process (ie: 'enter shipping address') are re-used from existing pages unrelated to the checkout process itself.

Related Patterns
Unknown

Monday, October 29, 2007

Update! New god conditions added to source repository!

Click the link to go to the god source repository - as of today, it looks like two of the three conditions I submitted for inclusion have now been added (the third being the mysql_failed condition, which may end up in some kind of auxillary gem or something, as previously mentioned...). So... look for these new conditions in god v0.6.0!

complex.rb

disk_usage.rb

Friday, October 26, 2007

New god Conditions

If you haven't checked out god yet as an alternative to monit or other system/software-monitoring tools, do yourself a favour and head over to that link for a while and then come back... it's an awesome little monitoring tool written in ruby that has all kinds of cool features, including event-based conditions that will activate as soon as a process dies instead of needing a periodic check, etc...

I won't repeat what's on their site - suffice it to say that it's a pretty nifty little piece of software, and highly useful. As such, I'm already using it in numerous places, and have written a few custom conditions to extend what's packaged with it.

At the time of writing, these have been submitted to the maintainers for future inclusion (current version is 0.5.0, so look for them in 0.6.0 hopefully!), but I've posted the files for my wonderful readers so they can start using them right away ;)

All three of these have been tested with god v0.5.0, mysql_failed has been tested w/ mysql.rb v1.24, and disk_usage has been tested w/ an installed df v5.3.0 (linux 2.6.21.3).

mysql_failed (if at all) might end up in an auxillary god gem, since it's application specific and has external dependencies. Likewise, I'm going to try and re-write disk_usage (this was a quick one that I just wanted to get working) so that it's not dependant on external programs (df), but figures it out some other way, although I suspect this may still have to be dependant somehow on the environment (ie: /proc or something).

A few usage notes:

1) These files should be put in <god-gem-install-root>/lib/god/conditions.
2) You must edit <god-gem-install-root>/lib/god.rb to require them (at the top of the file) in order to be able to use them.

complex.rb:

This condition lets you combine other conditions into compound conditions... ie: this AND (that OR these). Usage is fairly straightforward, it works more or less like any other condition (see example below). Complex conditions can be nested ad infinitum (to emulate parentheses in 'real' compound logic statements) and the 'this()' method can be omitted if it makes your code DRYer...


on.condition(:complex) do |c|
c.this(:memory_usage) do |c2|
...
end

c.or(:complex) do |c2|
files = %w(file1.txt file2.txt file3.conf file4.bak)

(0..3).to_a.each do |idx|
c2.or(:some_kind_of_file_based_condition) do |c3|
c3.filename = files[idx]
end
end
end
end


mysql_failed.rb:

This condition is intended to test all aspects of a mysql dependancy that your app may have (ie: connection and privileges... others to be added in the future if necessary/requested, perhaps a possible mysql version check?). It's fairly simple to use, the only things requiring examples are the default config setup, which allows you to skip specifying any/all config info when setting up subsequent instances of this condition, and the way of specifying privileges for the connection...


on.condition(:mysql_failed) do |c|
c.setUser('username')
c.setPass('password')
c.setHost('mysql.domain.com')
c.setDB('database_name')
c.setPrivs = {
'select' => %w(table_one table_two),
'delete' => ['table_three', 'table_four']
}
end

on.condition(:mysql_failed) do |c|
c.setHost('mysql2.domain.com')
end


Any info not explicitly set in secondary instances of the condition will inherit from the previously setup instance. So in the example above, the second mysql_failed condition will inherit the first mysql_failed's username, password, database name and privilege set to test for.

disk_usage.rb:

This one is simple enough to let the example do the talking...


on.condition(:disk_usage) do |c|
c.limit = 90 # percentage of partition that needs to be full
c.mount_point = '/usr'
end