Psychohistory, then, as defined by Isaac Asimov in his novel Foundation, goes something like this:

Branch of mathematics which deals with the reactions of human conglomerates to fixed social and economic stimuli; implicit in all these definitions is the assumption that the human conglomerate being dealt with is sufficiently large for valid statistical treatment.

In Asimov’s novels, psychohistory is used to predict the future on a large scale.  It cannot predict an assassination of a leader or the popularity of a celebrity.  These things are statistical anomalies.  Instead, psychohistory predicts the fate of a society, the social and economic trends of nations, that nation’s success or failure.

For now, though, predicting the future still falls firmly in the hands of frauds and science fiction authors.  However, given the Big Data paradigm shift, I’m not convinced it has to stay there.  Asimov has already predicted the miniaturization of consumer electronics, the modern internet as we know it, and humanoid robots (which I believe someday will move beyond the uncanny valley).  It isn’t hard to imagine integrating many years worth of behavior and census data with historical data and seeing patterns emerge and from there extrapolating, accurately, what is to come.

Psychohistory may be a pipe dream, but Asimov was a visionary and the future is a very exciting place

Is an empty string really a misspelled word?

(: I've declared the function in the nwnu namespace for testing purposes :)
declare function nwnu:number-to-seq($num as xs:integer)
as xs:integer* {
  if($num gt 9) then
    (
      nwnu:number-to-seq(xs:integer(math:floor($num div 10))),
      xs:integer(math:floor($num mod 10))
    )
  else
    $num
};

We’ll employ a recursive strategy to build our sequence.  First, we’ll check whether or not the number is greater than 9.  If the number is already a single digit, then we’ll just return it into the sequence since that is ultimately what we are adding to the sequence anyway.

Otherwise, we’ll add the the remainder of dividing the number by 10 with the modulus operator; this will isolate the right-most digit in the integer.  We’ll then recursively call our function with the rounded down result of dividing the input with 10.  So, for the initial input 1250, we will recursively call the function on:

1250 -> 125 -> 12 -> 1

Testing the function

I am using Xquery with the MarkLogic engine.  If you have a properly configured AppServer, you can test the function with the following script.

xquery version "1.0-ml";

import module namespace nwnu = "http://nelsonwells.net/xquery/number-utils"
at "/lib/number-utils.xqy";

let $seq := (13857, 1847471, 11183, 44819)
let $out := for $i in $seq
  return
    <item>
      <input>{$i}</input>
      <output>{nwnu:number-to-seq($i)}</output>
    </item>

return <items>{$out}</items>

Nothing like cracking the whip

var o = {
  mrn: '8675309',
  labs: [
    'blood bank',
    'microbiology',
    'toxicology'
  ]
};

console.log(JSON.stringify(o, null, '\t'));

However, there are any types of Mirth mapped variables, the JSON.stringify() method blows up on itself with a message similar to

DETAILS:    Java class "[B" has no public instance field or method named "toJSON".

This is how we fix it.

Consider the following code.  Assume that a variable has been mapped to mrn using Mirth's mapper transformer or a call to channelMap.put().  This code explodes with the aforementioned error.

var o = {
  mrn: $('mrn'),
  labs: [
    'blood bank',
    'microbiology',
    'toxicology'
  ]
};

Oddly, if we call logger.error($(‘mrn’)), the mapped variable shows up correctly.  Our assumption is that $(‘mrn’) would return a string if the mapped variable was in fact a string (as opposed to a javascript object, for example).  However, if we then call logger.error(typeof $(‘mrn’)), we get “object.”  This should be our first clue that not all is what it seems.

Since I am not a Java programmer, I searched the internet trying to find what “[B" represents.  I've found that [B is some kind of a byte array in Java, and thus Mirth or maybe even Rhino implements a mapped variable as a byte array.  When JSON.stringify() is called on our data structure, toJSON() is called on each individual element in the structure... since the java byte array [B does not implement toJSON(), an exception is thrown.

So, our goal is to convert the [B object into a javascript string since a javascript string does implement the toJSON() method.  My original instinct was to simply called toString() on the mapped variable, but that failed similarly.

I eventually found that explicitly constructing a javascript string with the new keyword would get the job done.  This is the working code for printing a data structure with a mapped variable.

var o = {
  mrn: new String($('mrn')),
  labs: [
    'blood bank',
    'microbiology',
    'toxicology'
  ]
};

logger.error(JSON.stringify(o, null, ‘\t’));

Identifying Necessary Information

ClicheSite obviously does not have all of it’s cliches listed on the home page.  It is technically feasible to begin at the homepage and follow links until we find something useful, but sometimes it makes more sense to take a manual step or two.  In this case, I identified alphabetic listings of phrases; each letter in the alphabet has its own page.  For example, http://clichesite.com/alpha_list.asp?which=lett+1 is a list of all phrases beginning with the letter A and http://clichesite.com/alpha_list.asp?which=lett+26 is a list of all phrases beginning with the letter Z.  So, it is clear that we can open and parse 26 pages to grab our phrases.

Next, we need to look at the pages’ HTML source and identify a way to grab the data we want.  I used Firebug to look at the HTML, but you can also just look at the source directly with your favorite browser.  In Firefox, for example, you can right click anywhere on the page and hit “View page source.”  In any case, here is a portion of the HTML we’re concerned with.

 <tr>
   <td valign="top" bgcolor="#FFFFCC">&nbsp;</td>
   <td valign="top" bgcolor="#FFFFCC">
     <span> <a href="content.asp?which=tip+1332">Acorn doesn't fall far from the tree, The</a></span>
   </td>
   <td valign="top" bgcolor="#FFFFCC">                      
     <span> United States</span></td>
 </tr>

<tr>
  <td valign="top" bgcolor="#FFFFCC">&nbsp;</td>
  <td valign="top" bgcolor="#FFFFEE">  
    <span> <a href="content.asp?which=tip+65">Actions speak louder than words</a></span>
  </td>                         
  <td valign="top" bgcolor="#FFFFEE">         
  <span> United States</span></td>
</tr>

Let’s examine what we have here.  The HTML shows us that we have two rows with 3 columns.  We see that the two phrases are in the second column wrapped in a span and anchored (linked).  One possible action we could take is to write an HTML parser or use a 3rd party library to parse the HTML and then drill down to that HTML node.  However, for this particular case, that solution is a shovel when all you really need is a spoon.  Instead, we’ll use a regular expression to grab the phrases… but how?

Notice that each phrase is surrounded by an anchor tag, and inside that anchor tag is the word “tip”.  That will be the basis for our regular expression.  The content between the anchor tag with “tip” in the href attribute will be our phrase.  We’ll see that regular expression in the next section.

The Code

Since the code is so short, I’ll let you look over it and familiarize yourself with it before I explain it.

var http = require('http');

var options;
var regex = /<a.*tip.*>(.*)<\/a>/g;

for(var i = 1; i <= 26; i++) {

  options = {
    host: 'clichesite.com',
    path: '/alpha_list.asp?which=lett+' + i
  };

  http.get(options, function(res) {

    // accessible to 'data' and 'end' events
    var data = '';

    res.on('data', function(chunk) {
      data += chunk.toString();
    })
    .on('end', function() {
      while((match = regex.exec(data))) {
        console.log(match[1]);
      }
    });
  });

}

First, we load the HTTP module.  We’ll use this to make HTTP requests to ClicheSite.  Then, we’ll define an options variable.  This is used as a parameter in the HTTP request and identifies the host and path (effectively, the URL) to hit.  Finally, we define a regex literal to identify the phrases.  Let’s break it apart piecewise.

<a   - less than followed by the letter a
.*   - followed by any character any number of times
tip  - followed by the letters t, i, p
.*   - followed by any character any number of times
>    - followed by a greater than
(.*) - followed by any character any number of times, and capture it
<    - followed by a less than
\/a  - followed by a forward slash and the letter a
>    - followed, finally, by greater than

We’ll match it globally.  Later, we’ll loop through the result of the regular expression and grab the captured phrase.

Now, since we’ve previously identified that we need to scrape pages where lett+ a number 1 through 26 appears in the URL, we’ll loop through that range.  We’ll populate that options variable we created earlier with an object with two properties, host and path.  The host is the domain we wish to hit with our HTTP request, and the path is basically everything past that point in the URL.  Notice that we’re appending our loop index to the end of the path.  That is how we’ll hit pages A through Z.

Now, we’ll make an HTTP request.  All we have to do is call the get method of the HTTP object. We’ll define a callback function with an HTTP result object as a parameter… we’ll register events on this res object to do the actual scraping.  In the .get callback, we define an empty string called data.  This will eventually hold the full HTML content of the page.

Next, we register two events on the res object.  The first event, “data”, is fired when data is received.  When that happens, we’ll just append that data to the data string we defined.  The next event, “end” is fired when the HTTP request has finished.  When that happens, we will have the full HTML document.  This is where we do the work.

In the callback to this event, we loop through the results of our regular expression with the following construct:

while((match = regex.exec(data))) {

The regular expression exec will eventually evaluate to null, a falsey value, and the loop will end.  In the body of the loop, we can access the phrase with match[1].  Match[0], of course, contains the whole matched expression; we just want the capturing group.  In this example, I simply print the phrase to the screen… you’ll likely want to do more to the data than that, though.

Pitfalls

One thing you’ll notice about this solution is that the results are not in alphabetical order even though we iterate alphabetically from A to Z.  This is because of Node.js’ event-driven, asynchronous nature.  In the body of the most broad loop, we make an HTTP request.  However, that request will most likely not finish before the we hit the next iteration of the loop.  Processing seed is much faster than network latency, after all.  In fact, when I was testing the script, I was often making 5-10 HTTP requests before the first one returned!  For this simple example, if you wanted a sorted list you could run the script and pipe to sort

node index.js | sort

More likely, you’ll want to store the results in a data structure and sort it or in a database so it can be sorted later.  This depends on context, of course.

Ironically, you also need to be wary of scraping pages heavy with Javascript.  If the contents of the page are dynamic, you may not have access to the data you actually want to get!

Final Thoughts

Why scrape web pages?  If you are feeling black hat, it is relatively simple to scrape for email addresses, phone numbers, and addresses.  In the real world, you may want to scrape your own web applications to ensure data validity and correct formatting of data.

In any case, please be nice!  Don’t break a website’s terms of use, and most certainly do not blame it on me!

Happy coding

The Invert Function

var invert = function (obj) {

  var new_obj = {};

  for (var prop in obj) {
    if(obj.hasOwnProperty(prop)) {
      new_obj[obj[prop]] = prop;
    }
  }

  return new_obj;
};

The function itself is fairly simple.  We’ll pass an object as a parameter; this is the object whose keys and values we’ll invert.  We then create a new empty object; this is the object that we will return with the inverted keys and values.  We do not operate on the object that we pass as a parameter because we do not want to change the original object.

First, we’ll loop through the properties of obj.  We’ll use hasOwnProperty to verify that the keys we invert were defined on the object itself and not up the prototype chain.  In the loop, we’ll create a new key from the value of the key in the original object.  We’ll assign the value of that key the property we’re currently looking at in our iteration.  That’s all there is to inverting the keys and values!

Of course, we’ll return the newly generated object.

Use Cases

Okay, so we can invert k, v pairs.  What can we do with that, though?  How about iterating over them and performing some kind of action on it?

var obj = {
  firstname : "nelson",
  middlename : "dane",
  lastname: "wells",
};

var name = '';

for(var value in invert(obj)) {
  name += value + ' ';
}

console.log(name.trim());

Maybe we want to know if a value exists in the hash map?

var obj = {
  firstname : "nelson",
  middlename : "dane",
  lastname: "wells",
};

if(invert(obj)['dane']) {
  //value exists in the hash
  console.log("Value exists");
} else {
  console.log("Value does not exist");
}

if(invert(obj)['terry']) {
  console.log("Value exists");
} else {
  console.log("Value does not exist");
};

Of course, there’s nothing preventing you from inverting the keys, saving the inversion in a var, and doing something with it later.  In fact, you may want to do that…

Shortcomings

There are some obvious shortcomings with this function.  First, every time we call it, we’re iterating over the object’s properties.  Of course, we’re not going to want to do this in a loop because of performance implications.  Instead, we would want to call the function once and assign the result to a variable, and operate on that object.

Another shortcoming is the hash map must have only one to one relationships.  If a value exists for two keys, when the object is inverted, we’ll lose a key in the process.  It should be trivial to add exception handling to the function or some other mechanism to handle this deficiency gracefully, so if anyone is interested in that and can’t do it themselves, I’d be happy to help.

We’ll also prefer non-nested objects.  An array, another object, or a function may not make a great key

We’ll first examine the naive approach and then the memoization method.  All in Javascript, and all with code samples…

The Naive Approach

The naive approach is very simple.  This is a snip / modification of the larger script, which you can see at the bottom of this post, but you will get the point.

function naive(n) {
  if(n <= 1) { return 1; }
    return n * naive(n - 1);
  }
}

In this function, the parameter n is the value of the factorial we wish to calculate.  In this recursive function call, our base case will be when the parameter is 1.  1! = 1, of course, so instead of recursing we’ll simple return the value.  If n is greater than 1, we’ll recursively call the function, returning the current value we wish to calculate multiplied by n – 1.

While this approach works, it is wasteful.  By calling naive(5), we calculate 5 * 4 * 3 * 2 * 1.  Instead of saving the value of naive(5) to be used the next time we call the function, we effectively discard it.  When we call, naive(7), we cannot reuse the value we found for naive(5).

The Memoization Approach

The memoization approach is only slightly more complicated.  Again, this code is a section of a larger script, so bear with me.

var lookup = {};

function memoized(n) {
  if(n <= 1) { return 1; }

  if(lookup[n]) {
    return lookup[n];
  }

  lookup[n] = n * memoized(n - 1);
  return lookup[n];
}

First, we create an empty object to serve as a lookup table for our factorial function.  This is where we’ll save the factorials that we’ve already calculated.  Our function has the same parameter and base case as our naive example.  The difference in the function is the lookup.  If the value of n is a key in our lookup table, then we simply return that value.  If the value isn’t in the lookup table, we’ll save it in the lookup table for later.  Now that we’ve found and saved the value of factorial n, we’ll return it to the caller.

If we first call memoized(5), we’ll calculate 5 * 4 * 3 * 2 * 1.  If we then call memoized(7), we’ll calculate 7 * 6 * 5!, the 5! being saved in the lookup table.  Much faster!

The Full Script and Observations

I said previously that the snippets above were part of a larger script.  In the process of writing this post, I wrote the functions as part of a Javascript object I also wrote a function to test the speed increase.  While not absolutely scientific, it should help you grasp the kind of increase you can expect over the succession of function calls.  The script was written and executed in node.js, but should work just as well in Firebug or even Rhino if you remove the #! script declaration.

For what it’s worth, on my machine node.js executed the test set for the naive method in ~5ms and the memoization method in ~1ms.  Firebug / Firefox 6.0 executed the test set for the naive method in ~14ms and the memoization method in ~1ms.  I think that says something about the speed of the V8 Javascript engine.

Feel free to use / modify this script for your own tests.  I hope that this example helps you speed up your code in the future.

#!/usr/local/bin/node

var factorial = (function() {
  var lookup = {};

  return {
    //show_lookup is mainly for debugging purposes
    show_lookup : function() { return lookup; },

    run : {

      naive : function(n) {
        if(n <= 1) { return 1; }
        return n * this.naive(n - 1);
      },

      memoized : function(n) {
        if(n <= 1) { return 1; }

        if(lookup[n]) {
          return lookup[n];
        }

        lookup[n] = n * this.memoized(n - 1);
        return lookup[n];
      }

    }
  };
})();

var caller = function(method) {
  var start = (new Date()).getTime();

  for(var i = 0; i <= 150; i++) {
    factorial.run[method](i);
  }

  for(var j = 150; j > 1; j--) {
    factorial.run[method](j);
  }

  return (new Date()).getTime() - start;
};

console.log(caller("naive"));
console.log(caller("memoized"));

Using http://whatismyip.com ‘s “automation” page, Perl, and cron, I am going to maintain a list of my external IP addresses.  Not only will this answer the original question, but I can also gather some additional information, such as how often my IP address changes and whether or not I can force my IP address to change by disconnecting my modem .

The logical steps

The method I am going to use is relatively simple.  First, I need a script that will fetch a page with my current external IP address.  I’m going to use Perl, but any scripting language will do.  Once the script has fetched my current IP address, I will append the IP address and the current date, separated by a comma, to a log file.  I want the two fields to be comma separated in case I ever want to run a script over the log it will be easy to grab the information I want.

Once the script is finished, I need a way to automate it.  I would like to do this once a day… you may want to do this less often or more often depending on your goal.  Luckily, this is simple to do with cron, the Linux utility to run scheduled tasks.

The code

First, we need to create the script.  I am going to call my script ip_logger.pl.  We need to declare that the file is a Perl script, give the usual safety directives, and load relevant libraries.

#!/usr/bin/perl
use strict;
use warnings;
use File::Slurp;
use LWP::Simple;
use Time::localtime;

I’ve loaded File::Slurp to ease reading and writing the the log file.  I’ve loaded LWP::Simple (libwww-perl) to fetch the page with our external IP.  I use Simple since we just need to make a GET request, we don’t need any additional features.  I’ve loaded Time::localtime to make generating a current date string simple and easy to read.

Next, we need to decide where we’re going to put our log file, and then actually make the request to the website to retrieve our external IP address.

# cron runs the script from your home directory,
# so give an absolute path
my $ip_addresses_filename = '/home/nelson/projects/perl/ip_logger/ip_addresses.txt';

# get current external ip address
my $content = get('http://automation.whatismyip.com/n09230945.asp');
$content = 'Could not pull ip address!' if !$content;

Let’s notice that we create a scalar called $content and populate it with the contents at the whatismyip.com ‘s automation page.  The page has no HTML, it only contains the IP address, so we don’t have to parse the IP address or do anything else.  We’ll just assign the returned string to the scalar.  Also notice that if the page fetch fails, we’ll put a string indicative of such in the $content scalar.

Next, we need to capture the current date.  I’m going for the month-day-year format.

# get the current date to also write to our file
my $month = localtime->mon + 1;
my $day = localtime->mday;
my $year = localtime->year + 1900;
my $current_date = "$month-$day-$year";

The last thing the script needs to do is actually write to the log file.  File::Slurp’s write_file function is smart enough to create the file if it doesn’t exist so we don’t need to guard against that.

# write / append it to the file
write_file($ip_addresses_filename, {append => 1}, $content . ',' . $current_date . "\n");

The cron

The script by itself will not do a whole lot of good.  Unless you want to manually run the script once a day, we need to schedule it to run automatically.  In shell, we will do crontab -e to open our crontab for editing.  Then, we’ll insert this line.

30 6 * * * /home/nelson/projects/perl/ip_logger/ip_logger.pl

I want to run the script at 6:30 AM every day.  As I said earlier, you may want to adjust the interval to something more appropriate for your situation.

The result

After a few days, I’ll have a log file that looks similar to this.

104.94.155.241,8-8-2011
104.94.155.241,8-9-2011
104.94.155.241,8-10-2011
104.94.155.241,8-11-2011
104.94.155.241,8-12-2011
104.94.155.241,8-13-2011

Happy automating!

MSH|^~\&|DDTEK LAB|ELAB-1|DDTEK OE|BLDG14|200502150930||ORU^R01^ORU_R01|CTRL-9876|P|2.4
PID|||010-11-1111||Estherhaus^Eva^E^^^^L|Smith|19720520|F|||256 Sherwood Forest Dr.^^Baton Rouge^LA^70809||(225)334-5232|(225)752-1213||||AC010111111||76-B4335^LA^20070520
OBR|1|948642^DDTEK OE|917363^DDTEK LAB|1554-5^GLUCOSE|||200502150730|||||||||020-22-2222^Levin-Epstein^Anna^^^^MD^^Micro-Managed
Health Associates|||||||||F|||||||030-33-3333&Honeywell&Carson&&&&MD
OBX|1|SN|1554-5^GLUCOSE^^^POST 12H CFST:MCNC:PT:SER/PLAS:QN||^175|mg/dl|70_105|H|||F

Notice the “Health Associates” segment? Obviously, this segment invalidates the message.  When this message is sent through a Mirth channel that expects incoming HL7, the message will error out when Mirth tries to parse it.  Here is how we can fix that.

The Solution

To fix this, we’re going to use a channel’s preprocessor script to append seemingly broken segments to the previous segment.  Since the preprocessor script is invoked before Mirth’s HL7 parser, we don’t have to worry about Mirth’s HL7 parser choking on it.

Instead of explaining the logic in a separate paragraph, I have left inline comments to explain what is happening.  I think, though, that it is fairly easy to follow.

Paste the following code into your channel’s preprocessor script.

The Code

// initialize a first_iteration flag.
// we don't want to add a carriage return
// before the MSH segment
var first_iteration = true;

// initialize the new message as an empty string
var new_message = '';

// loop over each line in the message
// by splitting on the hl7 standard carriage return
for each (var line in message.split(/\r/)) {

  // if it is a valid hl7 segment, we'll
  // append the new line to the new message
  if(line.match(/^[a-zA-Z0-9]{3}\|/)) {
    // see the first comment
    if( ! first_iteration){
      new_message += '\r';
    }
    new_message += line;
  } else {
    // if it isn't a valid hl7 segment,
    // we'll append it to the end of the
    // previous segment instead
    new_message += ' ' + line;
  }

  // it will never be the first_iteration again
  first_iteration = false;
}

//send the fixed message to the channel
message = new_message;

return message;