The reality of economics in tech innovation

There are things in the economies of tech businesses that scale well (mass production; agglomeration of labor) and there are things that don’t scale well. Innovation is not something that scales well, and so I will try and point out a few reasons why.

A few days ago HP announced that it is killing its tablet and spinning off WebOS. Perhaps the only surprising thing is that it happened so quickly; the HP tablet was only a few months old.

On the surface, you would imagine — as many people did — that a company as large, talented and wealthy as HP could actually pull off a device to compete with the Apple iPad. However, I think this was a poor supposition to have been made at all, for reasons I will go into in this post.

HP’s competitive advantage in the tech industry is that, up until maybe the last decade or so, it had the best engineers in the business. While that is not arguably true anymore, it is important to note that it never really had a significant competitive advantage in the mobile/tablet market. The mobile/tablet market is dominated by Apple, of course, whose competitive advantage is simply selling great design and great UI. Apple have the most market share because they are obviously the best at it. When was the last time you said “This HP device is really great, and really easy to use?” I will go ahead and give you the answer, i.e. “Never”. They had tried mobile before, with the iPaq smartphones, and they sucked.

While HP’s lack of competitive advantage in the mobile/tablet sector doesn’t mean the HP tablet was destined to fail, it sure had a huge mountain to climb if it was to be unseat the iPhone & iPad. Unfortunately for HP, they failed.

It is also worth noting that competitive advantages, like everything else in the world, are dynamic. Research In Motion (RIM) had a competitive advantage in building mobile devices that a) worked well with enterprise software and b) were very network-efficient. Now, neither are really true; as RIM employees are starting to see the writing on the wall. RIM failed to notice that customers started wanting different things: as the mobile market expanded to “regular” consumers (as opposed to corporate consumers), people want things like cameras and MP3 players and don’t care about “true” multitasking or many things RIM think they care about. But RIM is wrong, and it will kill the company if they don’t reverse their course.

Innovation can come out of nowhere, and it can disappear just as quickly. Perhaps the first great innovator in the mobile space was Nokia; now their products are openly mocked. They had the best technology at the time the technology was emerging; once the technology became common and cheap they were quickly displaced by other manufacturers who had better devices (Samsung, Motorola, etc.) Businesses cannot rest on their laurels and presume that their advantages will live forever.


BlackBoard & Oracle benchmarks on SSD storage

I spent Friday benchmarking two platforms against each other: our production BlackBoard instance (9.1 SP6) versus a development BlackBoard instance (also 9.1 SP6) that I spun up for this test. The unique thing about this test was that I put the development instance entirely on SSD storage.

Production instance:

  • BlackBoard 9.1 SP6 on VMware ESXi VM (Linux)
  • Oracle 10g R2 on HP BL465c G6 Blade (Linux)
  • BlackBoard on NetApp filer; 1 volume on a 45-disk RAID-DP aggregate
  • Oracle on local HP SAS RAID-1 disk set

Development instance:

  • BlackBoard 9.1 SP6 on VMware ESXi VM (Linux)
  • Oracle 10g R2 on VMware ESXi VM (Linux)
  • BlackBoard on FusionIO ioDrive SSD
  • Oracle also on (the same) FusionIO ioDrive SSD

With the exception of workloads (itself a significant difference, to be sure), I tried to keep everything else the same — same patches, same OS revision, same LDAP & SSL configurations, etc.

To test performance, I had a suggestion from Steve (@Seven_Seconds) at BlackBoard: hit the /webapps/login/ page with ab, the Apache benchmark tool.

The results were, well, strange. With each test of em running 5000 requests to /webapps/login/, here is a graph:

NetApp v SSD

You can see that the mean response times SSD (blue line) equals or out-performs the NetApp (red line) at every concurrency value tested. What is particularly interesting, though, is the incredibly slow rate at which the standard deviation of the SSD (purple bars) grows; compare that with the rate at which the standard deviation of the response times of the NetApp (green bars) grow.

Right now, I don’t have a clear answer as to why the deviations from the mean grow so differently. The NetApp filer has a significantly different workload to the SSD (the SSD has no workload other than this test), but the exponential growth of the NetApp’s standard deviation is something I’ll need to investigate further.

Last but not least, there were no SSD-specific tuning options put in place, nor where there any NetApp-specific tuning options. This was as vanilla as both installs could get in order to keep everything relatively comparable.

Deleting private messages queues in Windows 2008 Server

Ran into this little chestnut this morning. Windows 2008, in its infinite wisdom, does not let you delete private message queues; they are owned by the SYSTEM user and no one else. This was a roadblock to uninstalling a vendor’s HVAC platform, so here’s the way around it:

  1. Open Server Manager
  2. Click on the Features tree
  3. Click on the Message Queuing tree
  4. Expand the Private Queuing tree
  5. Right-click on the queue you want to delete and select Properties
  6. Click on the Security tab
  7. Click on the Advanced button
  8. Click on the Owner tab
  9. Click on your own account name, then click Apply
  10. Click on the Permission tab
  11. Click on the Add button and add yourself as a user
  12. Give yourself Full Control and click Apply
  13. Click OK
  14. Right-click on the queue you want to delete and select Delete

Et voilà! No more queue.

Using Splunk with BlackBoard

I have been using Splunk for a couple of years and am pretty happy with it. At the BbWorld DevCon 2011 some people asked me about it, so I thought I’d write up how and why I use it.


What is Splunk?

Splunk is a log aggregation & searching tool. Okay, so it does way more than just that, but that’s what we use it for. It lets you monitor pretty much anything: a file, a directory, a port, a socket, WMI, whatever; and collect the output. Then it lets you search the output. It is better at searching log files than anything I have ever used. It knows the difference between an Apache log file and an IIS log file and a Tomcat log file, so when you search them it’s aware of the correct formatting to use.

And then there are Apps.

How much does it cost?

Splunk is licensed by how many MB of data you process per day. There’s a free, lifetime license for <500MB/day users; anything over that and you have to pay. Quite a lot. The free version does have one caveat: it doesn't support authentication. So you'll have to firewall your Splunk server after the 30-day trial expires & the free lifetime license kicks in.

What does it run on?

Pretty much anything, it seems.


Installing Splunk on Linux

Because we’re monitoring files on our BlackBoard server, you should install Splunk on a central Splunk server, and also on the Bb app server itself. (If you want, you could do it standalone on your Bb server, but why bother?) Simply download the correct version and install it via RPM. Accept the license agreement, of course!

The default username for a new application installation is admin with a password of changeme.

Configuring the Splunk server

By default, it runs on port 3000; you may want to change that via this command:

sudo /opt/splunk/bin/splunk set web-port 80
sudo /opt/splunk/bin/splunk restart

Also, by default, Splunk does not accept data sent by other Splunk servers (or clients). You’ll need to enable that via this command:

/opt/splunk/bin/splunk enable listen 42099 -auth admin:changeme

(FYI, the default listening port is 9997.)

If you want to accept syslog messages (generally a good thing), run this command:

/opt/splunk/bin/splunk add udp 514 -sourcetype syslog -auth admin:changeme

Configuring your Splunk clients

This is a two-step process. You need to enable forwarding on your Splunk client, but you have an option to enable lightweight forwarding as well. The difference is, when you enable lightweight forwarding, it shuts down the Splunk GUI and removes some unnecessary services. This is a great thing to do on a Splunk “client” machine:

sudo /opt/splunk/bin/splunk enable app SplunkLightForwarder -auth admin:changeme
sudo /opt/splunk/bin/splunk add forward-server
sudo /opt/splunk/bin/splunk restart

Monitoring inputs

Splunk is all about inputs. You can get a great of what – and how – Splunk can index data here. In Linux, inputs conf is located in /opt/splunk/etc/system/local/inputs.conf. The syntax is pretty simple, here is an example of what I monitor:

host =


Note that you don’t have to monitor a series of files: you can monitor a whole directory, or files with wildcards, whatever. (On Windows, those paths would look like [monitor://C:\Logs\foo.log].)

Configuring the *NIX app

As mentioned above, Splunk is very extensible via its application collection. To collect the *NIX statistics (users, resource usage etc.) from a Splunk client, add this to /etc/hosts:         splunk      LOGHOST

On your Splunk client, add this to /etc/syslog.conf:

*.*                                                     @LOGHOST

On your Splunk client, restart the syslog daemon:

sudo /sbin/service syslog restart

What do I do now?

Save a search. Create a dashboard. Make an alert. Go forth!

Using local repositories with yum

One of the great advantages of Red Hat Enterprise Linux is the fact that it uses yum & that it ties into the Red Hat Network. That said, the RHN is subscription-based, and sometimes you either don’t want to license a new VM (if you’re tight on numbers) or you simply don’t care about getting security notifications. Trouble is, with yum, if the machine doesn’t exist in RHN, you can’t use yum at all.

(I suppose you could cheat and use a CentOS repository in your RHEL box, but that would be naughty! I bet it would work though…)

So I figured I would just use a RHEL DVD and create a yum repository out of that. While it obviously won’t give you security updates, it will let you use yum to install packages and resolve dependencies. Note that this also works with an ISO image instead of a physical DVD; simply see the note underneath the first step.

Mount your RHEL DVD

mkdir /media/cdrom
mount /dev/cdrom /media/cdrom

(If you’re using an ISO and not a DVD, substitute the above mount command with mount -o loop /path/to/RHEL_DVD.iso /media/cdrom)

Install createrepo

rpm -ivh /media/cdrom/Packages/createrepo* /media/cdrom/Packages/deltarpm* /media/cdrom/Packages/python-deltarpm*

Create a local repository

mkdir /media/repo
createrepo -o /media/repo /media/cdrom

Configure yum to use the local repository

Create a file called /etc/yum.repos.d/local.repo and populate it thusly:

name=Local ISO repository

If you’ve already installed the RPM GPG key, you won’t need that last line about gpgcheck.

Performance in strange places

Sometimes, you find performance improvements in places where you weren’t looking for an improvement. I will give an example: at a meeting last week, our webmaster was passing on complaints from users that their Content Management System was performing poorly. He had done a bunch of investigative work and concluded that the problem was likely an Active Directory problem. While that’s not exactly what he meant, it’s what he said, and our Active Directory administrator was a little annoyed.

What the Webmaster actually meant was that, when publishing several thousand files that live on a NetApp filer that is Active Directory-aware, there may be issues if you’re performing thousands of LDAP lookups serially. Which is a valid point: that can get computationally expensive. So, we got to talking about it, and another engineer mentioned nscd, the name services caching daemon — something I didn’t know existed outside NIS (i.e., legacy platforms).

Long story short, it made for an amusing (and satisfying) test.

/home#  time ls -al | wc -l

real    0m19.547s
user    0m3.729s
sys     0m1.700s

Almost 20 seconds to perform a lookup on an NFS-mounted directory that has almost 4,000 unique Active Directory users. That’s a lot of UIDs and GIDs! Now, we install nscd, and run the same test once to populate the cache.

Then we run the test again, with a populated cache. Care to see the results?

/home# time ls -al | wc -l     

real    0m2.241s
user    0m0.360s
sys     0m0.570s

From 19 seconds down to less than 3 seconds. Quite the improvement! Obviously you may need to adjust your cache parameters, particularly if this is a fairly inactive system most of the time but does have the occasional need for performance — in such a case, you may set your cache to expire unusually slowly. The lesson from this, though, is that sometimes you find gains when you weren’t looking. Or even in places you didn’t know gains could be found.

Using BlackBoard with Apache 2.2.x

At BbWorld/DevCon 2011 this year I attended a fantastic talk from José Manuel López Luján and Juan Francisco García García, both out of Tec de Monterrey in Mexico. The topic was Tuning BlackBoard 9 for high-demand environments, you can get their slides here. They had a lot of very interesting material in it, but one of the more fascinating things was that they’d literally pulled out Apache 1.3.x, which is bundled with BlackBoard, and inserted Apache 2.2.x in its place. Needless to say, I was very interested to see how they did it and try and replicate it in my environment.

Seeing as they’re using Solaris and I’m using Linux, I thought I’d give it a try based heavily on their notes (the beginning of which are here).

I’m copying and pasting these notes from my own Wiki, so the formatting might not be great. So, if you want a more complete version or have any questions, please let me know! You can find me on Twitter here; Jose is @jmanuel_ll and Juan is @jfgarcia11.

Again, thanks to them for their great idea and their documentation and support in getting this going on Linux. Rumor on the street is that Bb are pursuing modularization of the web server functionality on their own and we might see something like this in production some time soon!

These notes are all for RHEL5 64-bit Linux & BlackBoard 9.1.


yum -y install autoconf gcc make openssl-devel

# note we’re using an RPM of OpenSSL


Compiling options

Configuring httpd:

./configure --prefix=/usr/local/blackboard/apps/httpd \
--enable-ssl \
--enable-status \
--enable-autoindex \
--enable-setenvif \
--enable-alias \
--enable-so \
--enable-cache \
--enable-rewrite \
--enable-proxy \
--enable-proxy-http \
--enable-mem-cache \
--enable-file-cache \
--enable-headers \
--enable-usertrack \
--enable-expires \
--enable-deflate \
--enable-proxy-ajp \
--with-mpm=prefork \
--with-ssl=/usr \
--with-included-apr \
--with-mod_jk \

Optional configure flags

--enable-disk-cache --enable-mem-cache --enable-deflate

Compiling & installing

make && make install


Startup scripts

cp /usr/local/blackboard/apps/httpd/bin/apachectl /usr/local/blackboard/apps/httpd/
cp /usr/local/blackboard/apps/httpd/bin/apachectl /usr/local/blackboard/apps/httpd/

Edit the file to reflect the following changes:

# the path to your httpd binary, including options if necessary
# pick up any necessary environment variables
if test -f @@bbconfig.basedir@@/apps/httpd/bin/envvars; then
  . @@bbconfig.basedir@@/apps/httpd/bin/envvars


cp /usr/local/blackboard/apps/httpd/conf/httpd.conf /usr/local/blackboard/apps/httpd/conf/

Second, change all instances of /usr/local/blackboard to @@bbconfig.basedir@@

Now, have the file reflect the following:

ServerRoot "@@bbconfig.basedir@@/apps/httpd"
Listen @@bbconfig.unix.httpd.portnumber@@

Now, add the following:

LockFile @@bbconfig.basedir@@/logs/httpd/httpd.lock
PidFile @@bbconfig.basedir@@/apps/httpd/logs/
Timeout 300

Add the following:

KeepAlive @@bbconfig.unix.httpd.keepalive@@
MaxKeepAliveRequests 100
KeepAliveTimeout @@bbconfig.unix.httpd.keepalivetimeout@@
MinSpareServers @@bbconfig.unix.httpd.minspareservers@@
MaxSpareServers @@bbconfig.unix.httpd.maxspareservers@@
StartServers  @@bbconfig.unix.httpd.startservers@@
MaxClients @@bbconfig.unix.httpd.maxclients@@
MaxRequestsPerChild @@bbconfig.unix.httpd.maxrequestsperchild@@

Add the following:

# Autoindex config

IndexOptions FancyIndexing
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*
AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core
AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^
DefaultIcon /icons/unknown.gif
ReadmeName README
HeaderName HEADER
IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t

Add the following:

BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0

Add the following:

DirectoryIndex index.html

Add the following:

Include conf/proxy_ajp.conf

(See below for populating proxy_ajp.conf.)


        ProxyRequests Off
        # Proxy Exception 
        ProxyPassMatch ^(/img/.*)$ ! 
        ProxyPassMatch ^(/special/.*)$ !
        ProxyPassMatch ^(/server-.*)$ !
        # remove TRACE support
        ReWriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
        ReWriteRule .* - [F]

        # always keep the host header
        ProxyPreserveHost On
        LogLevel warn   
        ErrorLog /usr/local/blackboard/logs/httpd/error_ajp_log
          Order deny,allow
          Allow from all
        ProxyPass / ajp://localhost:8009/               

Note that the ProxyPass parameter specifies port 8009, which is different to port 8007 in the Tec de Monterrey documentation. This is because our file specifies:


You can find full documentation for the mod_ssl config file here: [ here]. You’ll need to:

cp /usr/local/blackboard/apps/httpd.working/conf/ssl.conf /usr/local/blackboard/apps/httpd/conf
cp /usr/local/blackboard/apps/httpd.working/conf/ /usr/local/blackboard/apps/httpd/conf
mkdir /usr/local/blackboard/apps/httpd/conf/certs
cp /usr/local/blackboard/apps/httpd.working/conf/ssl.crt/server.crt /usr/local/blackboard/apps/httpd/conf/certs
cp /usr/local/blackboard/apps/httpd.working/conf/ssl.crt/server.key /usr/local/blackboard/apps/httpd/conf/certs

Now you’re ready to make the following changes to the .bb file:

LoadModule ssl_module libexec/
SSLCertificateFile @@bbconfig.unix.ssl.certificatefile@@
SSLCertificateKeyFile @@bbconfig.unix.ssl.certificatekeyfile@@
SSLLog @@bbconfig.basedir@@/logs/ssl_engine_log
SSLRandomSeed startup builtin
Listen @@bbconfig.unix.httpd.portnumber@@
Listen @@bbconfig.unix.httpd.ssl.portnumber@@
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl
SSLProtocol All

#This line is needed to make SSL and IE work correctly.
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

#Inter Process Session Cache
SSLSessionCache dbm:@@bbconfig.basedir@@/logs/httpd/ssl_cache
SSLSessionCacheTimeout 300

SSLEngine On
ProxyRequests Off
RewriteEngine on
RewriteRule ^/$ /index.html
#pass requests for / to /index.html
# We do not support perl servlets anymore
JkMountCopy On

Note that these changes reference the following from



That’s it for now — I’ve gotten this working in Bb 9.1 SP6 but I haven’t had the time yet to perform any significant benchmarking. More to follow when it comes…