Hiatus

For the next month or so expect postings here to be infrequent. I'm more likely to post short snippets on Twitter or Google Buzz during this time.

/dev/tty

/dev/tty	==
	Any questions?

Uncanny Friending

There is an urban legend that Eskimos have many different words for snow. The truth is the Aleut languages have about as many words for snow as does English, but allow descriptive suffixes to be attached to any word to form countless variations.

Consider the English words we use to describe human relationships, and the distinctions they convey in meaning:

sister	stepsister	half sister
significant other	fiancée	spouse
friend	just friends	friend with benefits
peer	coworker	colleague
mother	stepmother	godmother

We use adjectives to add huge amounts of information in a single word. "fiancée" conveys one meaning, that of a beloved person. "current fiancée" conveys an entirely different meaning, a disposable relationship given a label for convenience.

Now consider the words we use to describe relationships in social networks:

friend	friend	friend
friend	friend	friend
friend	friend	friend
friend	friend	friend
friend	friend	friend

Why do we find this unsatisfying? I believe it is a corollary to the Uncanny Valley effect in robotics and computer games: "friend" is close enough to the real description of the human relationship that we find it unsettling. If the term were more inhuman, less shaded with meaning, it would not be so maddening.

The term "like" has a similar problem: who wants to like something unpleasant or unsavory? Clicking "like" is meant is to express interest, but the terminology is close enough to the real intention to be maddeningly imprecise.

I also suspect this vaguely unsettling feeling will resolve itself in a few more years online: the words friend and like will simply lose all meaning. We'll know this has been achieved when people stop using air quotes to distinguish online friending versus real life friends.

This genesis of this musing came via an insightful tweet by Marshall Kirkpatrick:

told my wife that google "results from your social circle" showed me because we are friends. she insists we are more than that. true :) less than a minute ago via TweetDeck Marshall Kirkpatrick
marshallk

Death of Copper Predicted. Film at 11.

Every handful of years we ratchet up the Ethernet link speed: from 10 Mbps to 100 Mbps in the early 1990s, to 1 Gbps in the mid 1990s, to 10 Gbps in the early part of this century. 40 Gbps is the next target. At the 1 Gbps and 10 Gbps transitions naysayers maintained that copper cables would never be able to meet the required signaling rates and that optical would prevail. The same doubt is now being voiced about 40 Gbps.

During the 1 Gbps and 10 Gig transitions, optical media became available several years before copper, and then the initial 10 Gig copper specs were limited to patch cable distances of 10-15 meters. 40G will repeat the story with optical products already available, substantially before copper. Nonetheless I'd wager 40G copper transceivers will eventually appear in some form.

Yet this time, optical will win. Not because of the technology or limitations of copper wire, but because of economics. Economics used to be in copper's favor: simple install and no expensive lasers. Copper could ride the silicon technology curve, throwing ever more DSP power at the problem. Times have changed: cat6a and cat7 cabling is as difficult and expensive to install as fiber, and solid state laser components allow optical transports to ride the silicon technology curve.

• Like fiber, cat6/7 cables have a minimum bending radius. Pull too tight and the cable can no longer handle long distances.
• Like fiber, cat7 does not tolerate being stretched. Stretch a 100m cable by a centimeter and its performance suffers.
• Even padded cable staples put too much pressure on the cable. cat7 must run in a tray or conduit, and the bulky shielding means fewer of them will fit.
• cat7 cables are very sensitive to connectorization. The crimp tool you used for cat5e won't do.

The other problem with copper cables is that they are made of copper, an actively traded commodity. The chart below shows the raw material cost of copper over the last century, normalized to the US Dollar in 1998. During much of the late 1990s and early 2000s copper was cheap by historic standards. In the last few years the commodity price has trended back up due to demand, without a matching increase in new supply. If there is a natural ceiling for copper pricing where the market will seek alternatives, we do not appear to have hit it yet.

(data source: US Geological Survey)

I'm not predicting that 40 Gig copper transceivers will be impossible. On the contrary, I suspect there will be two solutions brought to market: a very short reach spec using RJ45 patch cables, and a 100m spec which imposes more painful requirements like cat7a/cat8, use of multiple cables, and electrically better connectors (presumably also manufactured, not connectorized on site). These products will eventually appear, substantially lagging optical product availability.

I simply suspect that the economics no longer work in coppers favor: patch cables from one side of the rack to the far corner will be long enough to have to worry about install quality. If the pressure from zip-ties fastening the cable to the rack threaten the operation of your network, you're better off using fiber.

The genesis of this post came as a comment on Stephen Foskett's excellent Pack Rat blog. It is an excellent resource, highly recommended.

Privacy vs Voyeurism

Much has been written about privacy online. When Pandora reveals our friend's music tastes it makes us slightly uncomfortable, even if we enjoy the new music suggestions which result. When our friends can unknowingly reveal information about us, we find it disturbing. Facebook privacy currently dominates the discussion, but the trend of all online activity has been more sharing and less privacy.

I use foursquare, which allows friends on that service to see your location when you checkin. Earlier this week a friend checked in to the Lucile Packard Children's Hospital.

A checkin notification is devoid of context; there was no indication if it was routine or emergency. Certainly if one had just rushed a child to the hospital one wouldn't bother checking in... but what about hours later? What about an extended stay, after initial panic subsides? Where detail is lacking, the mind fills in possibilities. After thinking about it for a while, worry overcame reservation and I sent email asking if there was anything I could do to help.

As it happens, the visit was completely routine.

It felt weird, asking if everything was ok. I was acting on the basis of information which even just a couple years ago would not have been available to me. Back then I would only have known if he'd informed me directly, and in that context asking if I could help wouldn't have seemed even slightly awkward.

Even if it hadn't been a routine visit, even if there had been help I could provide, reaching out on the basis of a foursquare checkin would have still felt weird. Why is that? I think it is a form of guilt, as using social media in this way feels a bit like voyeurism. In this case it was information the person had chosen to share by explicitly checking in on foursquare, but down in the subconscious it is still equated to clandestine spying.

As online privacy recedes, I think we're all going to be experiencing this feeling more often.

---

Thoughts on Sharing

Society does not inherently guarantee our privacy. It never did. The privacy most of us enjoy is actually anonymity. Celebrities struggle greatly to keep any portion of their lives out of public view; when you discard anonymity, privacy tends to go with it. As communications technology improves, the bar to achieve a degree of celebrity is lowered. I suspect the further back in history you go the difference will be the geographical radius of ones renown, not its impact.

We're rushing into a world where a huge percentage of the population will experience the advantages and disadvantages of losing anonymity in their daily lives.

• The eCommerce site will know your approximate net worth.
• The customer service response will be finely tuned to the likelihood your displeasure could damage their business.
• Product companies will assemble marketing lists of people who are statistically more likely to buy their product. Not by placing ads in venues they are likely to frequent, but by targeting them directly.
• When I look for a dance class for my daughter, I'll know if her friends are already enrolled somewhere without having to ask them.
• Insurance as we know it today will fade away, uncompetitive. It will not use actuary tables, it will be essentially an auction based on a tailored risk profile.

We might recoil from this, but I suspect it is not something which can be stopped. The technology has reached the point where these things are feasible, and there is a huge economic incentive to do so. A concerted effort to stop it results in the technology being less visible, not absent.

Update: Louis Gray, the friend whose hospital checkin triggered this musing, has posted some thoughts on location-based services and what information we make available to others.

Safe Food Design: Hot Dogs

The American Academy of Pediatrics released a policy statement calling for the redesign of foods which pose a choking hazard to children. Among the foods listed are hot dogs.

I humbly present my proposal:

Deep Pockets

In the tech industry, how often do we hear this?

• "Great technology, they just didn't market it well."
• "Its a shame to see that product die.
• "They need somebody with deep pockets to see it through."

Of course this brings us to Palm, acquired by HP for $1.2 billion. Brian Humphries, an HP executive in business development, reportedly said: "Our intent is to double down on webOS." Palm managed to find their deep pocketed benefactor. Now we get to watch what happens.

This is the second time a savior has swooped in for Palm. In the early 1990s before the PDA had really established itself as a category, Palm nearly ran out of money. Its VCs were unwilling to put in more, but Palm was not generating enough revenue to operate. The company was purchased by US Robotics, which was later purchased by 3Com. Palm operated successfully for many years after that first brush with death.

We'll see what happens from here. HP likely believes that by owning the complete system, from hardware to OS to applications, they will be able to deliver compelling products and compete successfully with the iPhone. Time will tell.

HTML5 is Hard, Lets Go Shopping!

I just wanted to embed two short audio clips in a web page. Just two little "play" buttons. Thats all. I started with a Flash player, "borrowing" one used by Google Reader:

<embed
  type="application/x-shockwave-flash"
  src="audio-player.swf?audioUrl=myfile.mp3">
</embed>
 

This worked fine, but its a brave new world. I decided to use HTML5's <audio> tag, falling back to the Flash player if <audio> is not supported. This results in:

<audio src="myfile.mp3" controls autobuffer>
  <embed type="application/x-shockwave-flash"
    src="audio-player.swf?audioUrl=myfile.mp3">
  </embed>
</audio>
 

Loaded it into Chrome, it looks nice and plays fine. Life is good. I feel like a real web-enabled kindof guy. Before posting I try it in Firefox... whoops, it doesn't play. Firefox 3.6 doesn't handle MP3 files, most likely due to patent issues. So Firefox has an empty gray box with a little "X" through it.

In fact there is no single audio format supported by all common browsers. Supplying both MP3 and Ogg Vorbis is recommended for maximum compatibility. Next step: re-encode the audio and supply multiple formats. Ogg has to be first, because apparently if Firefox cannot play the first format it does not try subsequent ones.

<audio controls autobuffer>
  <source src="myfile.ogg"/>
  <source src="myfile.mp3"/>
  <embed type="application/x-shockwave-flash"
    src="audio-player.swf?audioUrl=myfile.mp3">
  </embed>
</audio>
 

This sortof works. Not really, but sortof. Chrome doesn't seem to like the Ogg file and plays static for the last half second instead. It probably doesn't play in Opera, which considers the src attribute of the audio tag to be mandatory. I have no idea what IE will do. At least Firefox is happy.

To get an audio tag which will work in all browsers, it appears I have to use JavaScript. Detect the capabilities of the browser, assemble an audio object in the DOM which meets their various requirements and bogosities, and hope for the best.

It shouldn't be this hard. Really, it shouldn't. It appears that as with nearly everything else in the modern web, the HTML5 media tags will be buried behind APIs in our Javascript frameworks to work around browser differences.

Another Long Hidden Easter Egg

The Colony is an early 3d first person shooter. David Alan Smith began development in 1984 using an original 128k Macintosh. The game shipped in 1988. The version I played was black and white, though apparently a later update colorized it for the Macintosh II and Amiga.

The Colony is a fascinating game, for the daunting effort required to get good 3D performance on such early systems. For example the game engine does not use radians or degrees to compute angles. It settles for 256 discrete pseudo-degrees, allowing sin() to be implemented as a lookup table. cos() uses the same table by adding 64 to the desired angle, shifting sin by 1/4 wavelength. David A. Smith wrote about the development experience several years ago, and posted two videos of the game in action.

The game is also notable for its almost sadistic sense of humor. One of the puzzles is the Monolith: two connected rooms, painstakingly modeled after the small apartment at the end of 2001: A Space Odyssey. There is no exit. Once you find yourself in the room, there is no way to leave it. I saved a game in that room lo these many years ago, but unfortunately cannot load it due to a copy protection system of looking up a code in the long-lost manual. This screenshot comes from the second of the aforementioned Youtube videos instead.

Poking around the game's implementation reveals another little joke from its developer. Sound effects are stored as sfil resources in a file named Zounds. One of the resources, "test," is nowhere used in the game.

(MP3) (Ogg)

Sounds vaguely human, doesn't it? Played backwards and slowed by 2x, it is:

(MP3) (Ogg)

I first stumbled upon this sound effect in college. I wish to note that I was living in a dorm 800 miles removed from my parent's basement at the time.

Note: the sound effect is copyrighted, by Mindscape and David A. Smith. It is two seconds long and clearly not material to the game, since it doesn't ever play. I believe this blog post constitutes fair use, as commentary.

eWhere? eWorld!

Did anybody else have an eWorld account, or just me?

It never really took off as a place for those-of-us-who-drink-the-Apple-coolaid to congregate. Mostly it turned into a way to download System Updates before anyone else, so as to lord it over them later. "Oh, you're still running MacOS 7.6.1? 7.6.2 is waaaaaaayyy better."

Code Snippet: hash_map

hash_map is not part of the current C++ STL, but is universally adopted as an extension. Using hash_map is simple when using built-in primitives like int or char*. Attempting to use user-defined classes as a key is somewhat more difficult, as the following example demonstrates:

class example1 {
 public:
  example1() {};
  ~example1() {};

  uint8_t name_[8];
};

typedef __gnu_cxx::hash_map<example1,int> Example1HashType;
Example1HashType hash_map1;

int main(int argc, char **argv) {
  example1 e1;
  hash_map1.insert(std::make_pair(e1, 1));
}

When compiled with gcc 4.2 this results in:

/usr/include/c++/4.2/bits/stl_function.h:200: error: no match for ‘operator==’ in ‘__x == __y’

/usr/include/c++/4.2/ext/hashtable.h:595: error: no match for call to ‘(const __gnu_cxx::hash<example1>) (const example1&)’

hash_map requires two things: a hash function, and an ability to compare equality. To use a class as a key you need to provide a hash<> template specialized for the class in question. You also need to implement an equality operator.

class example1 {
 public:
  example1() {};
  ~example1() {};

  bool operator==(const example1 &other) const {
    return (memcmp(name_, other.name_, sizeof(name_)) == 0);
  };

  uint8_t name_[8];
};

namespace __gnu_cxx {
template<> struct hash<example1> {
  size_t operator()(const example1& k) const {
    size_t hashval = 0;
    for (int i = 0; i < sizeof(k.name_); ++i) {
      hashval = 5 * hashval + k.name_[i];
    }
    return hashval;
  }
};
}  // namespace __gnu_cxx

typedef __gnu_cxx::hash_map<example1,int> Example1HashType;

int main(int argc, char **argv) {
  example1 e1;
  Example1HashType hash_map1;

  hash_map1.insert(std::make_pair(e1, 1));
}

This works, but what if we cannot add an operator method? For example, perhaps the class is in a library which we cannot modify, or is created by a code generator. Consider this case where key is a simple struct with no member functions. This code fails to compile, due to lack of "__x == __y"

struct example2 {
  uint8_t name_[8];
};

namespace __gnu_cxx {
template<> struct hash<example2> {
  size_t operator()(const example2& k) const {
    size_t hashval = 0;
    for (int i = 0; i < sizeof(k.name_); ++i) {
      hashval = 5 * hashval + k.name_[i];
    }
    return hashval;
  }
};
}  // namespace __gnu_cxx

typedef __gnu_cxx::hash_map<example2,int> Example2HashType;
Example2HashType hash_map2;

int main(int argc, char **argv) {
  example2 e2;
  hash_map2.insert(std::make_pair(e2, 1));
}

As this is C++, the solution will of course involve more templates. hash_map does not directly invoke "x == y," it uses an equal_to<> template. The "__x == __y" compiler error is from the template. We can provide an equal_to<> specialization instead.

struct example2 {
  uint8_t name_[8];
};

namespace std {
template<> struct equal_to<example2> {
  bool operator()(const example2& x, const example2& y) const {
    return (memcmp(x.name_, y.name_, sizeof(x.name_)) == 0);
  }
};
}  // namespace std

namespace __gnu_cxx {
template<> struct hash<example2> {
  size_t operator()(const example2& k) const {
    size_t hashval = 0;
    for (int i = 0; i < sizeof(k.name_); ++i) {
      hashval = 5 * hashval + k.name_[i];
    }
    return hashval;
  }
};
}  // namespace __gnu_cxx

typedef __gnu_cxx::hash_map<example2,int> Example2HashType;
Example2HashType hash_map2;

int main(int argc, char **argv) {
  example2 e2;
  hash_map2.insert(std::make_pair(e2, 1));
}

Extending hash_map to custom key types makes it useful in far more situations.

Facebook Developer Humor

Facebook caps the maximum image size to conserve storage space, or bandwidth, or both. I attempted to upload an image substantially larger than this, and received this error.

I suspect the wording is a joke by Facebook developers, "You can never be too tall or too skinny."

Bending the Rules

As of yesterday the iPhone Developer Program License reads:

3.3.1 - Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

The way the new requirement is phrased is interesting. Adobe's Packager for iPhone compiles Flash code to native ARM instructions. Assuming the Adobe tool can produce position independent code, one might consider something sneaky.

void run_disassembled_flash_binary()
{
  asm volatile("ldr r3, [pc, #192]\n\t"
               "mov r1, #3\n\t"
               "str r3, [sp, #32]\n\t"
               "mov r3, #0\n\t"
               "str r3, [sp, #4]\n\t"
               "str r3, [sp, #8]\n\t"
               "mov r2, #24\n\t"
               "mov r3, #48\n\t"
               "add r5, sp, #12\n\t"
               "str r0, [sp, #24]\n\t"
               "str r0, [sp, #0]\n\t"
               "add r0, sp, #12\n\t"
               ...

I make this suggestion in jest. Apple does allow embedded ARM assembly in iPhone apps, but an application consisting of nothing but opcodes produced by a foreign programming environment will not resemble an app compiled from more conventional source code. I very much doubt it would pass the App Store approval process. Though phrased as a technical requirement, section 3.3.1 of the iPhone Developer License is really a business imperative.

Simple Checksums Considered Harmful

Lets talk about iSCSI for a moment, as a launching point for a discussion about data integrity. iSCSI relies on CRC32 to catch data corruption. CRC32 is a good fit for this purpose, but most previous uses of it had been confined to very low levels of the system and implemented in hardware. iSCSI uses CRC32 way up in the protocol header, where it is generally computed in software. The overhead of computing the CRC is one reason why so many hardware offload adaptors were developed for iSCSI.

Intel recently released a whitepaper describing how they achieved 1 million iSCSI operations per second. One fascinating tidbit is that the CRC32 is no longer a bottleneck. The Nehalem architecture includes an instruction to compute it directly, as part of SSE 4.2. The new instruction is described in the Intel64 and IA-32 Architectures Software Developer's Manual Volume 2A: Instruction Set Reference, A-M. It is on page 3-221 of the December 2009 edition; search for CRC32 in later editions.

CRC32 r32, r/m8	Accumulate CRC32 on r/m8
CRC32 r32, r/m16	Accumulate CRC32 on r/m16
CRC32 r32, r/m32	Accumulate CRC32 on r/m32
CRC32 r64, r/m8	Accumulate CRC32 on r/m8
CRC32 r64, r/m64	Accumulate CRC32 on r/m64

Thats it. You load words from memory and hand them to the CRC32 instruction. If you were already making a pass over the data for any reason, the CRC calculation is free. Table-driven CRC generation implementations were already fast, but this is even faster.

What does this mean? I think it means weak checksums should no longer be used for anything. Applications which care about data integrity moved to MD5 or SHA1 years ago, but you still see specifications in other contexts written to use Adler-32 or even the venerable 16-bit TCP checksum. Its not appropriate to use these any more. Server CPUs can compute CRC32 for free, and embedded CPUs have long included CRC32 calculation in DMA engines.

Peepshi

Just before Easter weekend Serious Eats published an excellent recipe for Peepshi. I highly recommend it. Their presentation is much better, I am just an amateur at food pr0n.