Experiential versus Informational Search

Yesterday Louis Gray posed a set of questions which seem like they should be easy to answer, but aren't. While reading it one of the questions stood out, for obvious reasons.

"2. When was the first time Denton Gentry left a comment on my blog?"

Louis had sent that question via email earlier in the day, and it turned out to be very difficult to answer. My profile on disqus.com shows comments going back to July 2009. That should be definitive, but unfortunately isn't correct as manual checking had already turned up earlier comments. In the end Louis answered his own question by searching his email for Disqus notifications. It was a full year before the first comment shown on my profile page. The other questions were similar: find the first citation. From a technical perspective, it should be easy to answer questions like this as all of the information is available. That it isn't easy is a reflection of economic reality. There is infrequent demand for it.

I'd like to flip it around, though: why is email able to answer questions like this? You can search email and sort it by date. You can find emails around a particular time. You can find emails which happened at about the same time as some other event which is unrelated, but intertwined in your memories. Why is email structured this way?

I suspect this is a reflection of human psychology. Email is information which we personally experienced. It exists in our own memories, albeit dimly or imperfectly. When we go to search for it, we're searching as an extension of our own memory. Its Experiential search, not Informational, and email services which don't match our expectations in this regard get less traction. This is also what makes services like Evernote so useful, letting us organize and search arbitrary information Experientially.

In comparison when searching for something we never personally experienced we're looking for information which we know must exist, and we just need to find it. Search engines are designed to this expectation.

The disconnect occurs when we want an Experiential search over an Informational dataset. Organizing arbitrary information in a way which maps to what we'd expect had we personally experienced it is an unsolved problem. It has been a rich field of speculation in science fiction, as authors have postulated implanted memories and neural interface.

Will there be developments in this area? Clearly there is at least some demand, as LexisNexis can answer such queries for the subset of publications they handle. Its something we'll need to work on if we're going to make the world even more like science fiction.

Content-Type: joke; genre=bar/packet

1. An IPv6 packet wants to walk into a bar, but can't cross the street to get there.
2. An STP packet closes the bar so the bar next door can continue to operate.
3. A VRRP packet gets confused about which bar it is in. Then it sets both bars on fire. Nobody knows why.
4. A BGP packet gives you a list of every bar in the world, everywhere. China crosses 25% of them off the list.
5. A multicast packet bar hops.
6. A forlorn BootP packet sits in a corner crying into its beer. Nearby, a DHCP packet is the center of attention.
7. An SCTP packet is stopped at the doorway and sent away.
8. An anycast packet doesn't really care which bar its in.
9. The LLC/SNAP packet has such a funny accent the bartender doesn't understand it.
10. An LACP packet comes in through the window, just checking that all of the paths work.
11. A large group of ARP packets storm in and make a general nuisance of themselves.
12. The bar hasn't been allowed to segregate customers by VLAN since the 1960s.
13. An IP fragment walks into the bar. The bartender makes it wait to be seated until the rest of its party arrives.
14. A kerberos packet walks into the bar, but is confused by the clock being 6 minutes fast. The NTP packet hands it a drink, and all is well.
15. An SFlow packet gossips with the bartender about all of the other packets.
16. The rest of the packets look pityingly at the old IPX packet.
17. The strict source routed packet tries to get to the bar but ends up at the dry cleaners instead.
18. The GRE packet carries another packet into the bar, sets it on the barstool, orders it a drink, and goes back to help another.
19. The TTL=1 packet barely makes it to the bar.
20. The IP Options packet arrives late. It had to take the slowpath.

I suspect many of these will only be funny to about 3 people in the world. If you are not one of them, my deepest apologies.

Home Made Cable Spaghetti

I wrote some thoughts for a colleague about home installation of a rack for computer equipment. Much of it is generally applicable for anyone considering such a thing, presented here for your edification and bemusement.

General

• buy extra rack screws, maybe 20. The cheap ones strip easily, and nothing sucks harder than getting a new system in only to discover you've run out of rack screws.
• Get an electric screwdriver if you don't already have one.

Physical Installation

• Bolt the rack to the floor and the ceiling. Otherwise an earthquake which doesn't otherwise damage the house can rip the rack out of the floor. The rack itself will be shorter than ceiling height, you get an extended brace to bolt to the ceiling. You want to bolt it to joists, not just drywall.
• There are also racks made to bolt to the wall rather than free standing floor to ceiling. These tend not to be as deep front to back, so it impacts the gear you can put in it. Also they can be an airflow problem if the gear vents front to back.
• Perhaps obviously, when filling the rack start from the bottom and put the heaviest gear at the very bottom. A top-heavy rack is a disaster.
• The industry never settled on whether airflow is front to back or side-to-side. You'll find equipment with both layouts. With one rack it doesn't particularly matter, and you can mix them. With multiple racks it matters a lot.
• The industry also never settled on whether rack ears go at the very front of the equipment or the midpoint. Front is most common, to accommodate boxes of differing depths. Lots of gear has threaded screwholes at front and midpoint, just be consistent.
• 19 inch racks are by far the most common, but be aware that 17 inch and 23 inch both exist. They will be well-labelled in catalogs as they are not common. If you buy second hand, bring a tape measure.

Cable Management

• Spend as much time thinking about cable management as you do about how to rack the machines. Otherwise you end up with a beautiful rack covered in cable spaghetti.
• Its customary to put the network switch at the top of the rack, because gravity makes cable management easier. However its not essential, and you can put it anywhere you like.
• There are cable trays with removable fronts made to bolt vertically to the side of the rack or between adjacent racks. HIghly recommended.
• Label both ends of each cable. Label them in a way which will still make sense in a few years when you replace these machines and have forgotten everything about the construction.
• Avoid labeling cables according to their destination within the rack. That changes over time, relabeling cables is a pain.

Mechanical Denial of Service

Mechanical Engineers need to worry about denial of service attacks too.

All in Good Time

Current directives:

1. Learn to dance.
2. Kill all Humans.

The ordering is important.

(via @mattcutts)

From a Lawnchair Overlooking the Bot War

In the last several days I've noticed a large increase in the rate of bots following me on twitter. It went from perhaps one a day to dozens. The bots are following 100-200 people, and have always sent zero tweets. They sometimes have an avatar picture, and their bio always sounds convincing. The avatars and bios are probably harvested from real Twitter accounts, and sometimes they get the default avatar. They appear to avoid any bios with a link.

The email from Twitter tells you what client was used to follow you. They have cycled through various third party clients, and then Twitter's mobile web page. Most recently the email contains no mention of the client, and I don't know what that means.

What is interesting is that this influx of follow bots never appear to send any tweets. An entirely different herd of bots has started spamming via @-replies, with a link purporting to offer a free iPhone. The bots which send the spam follow zero people.

I wonder: are the botnets now fielding offensive and defensive teams? By this I mean using the offense to send spam, and watch for twitter users which block them. They can check whether the bot can still see the tweets of the users it has spammed. Users who react by blocking are likely reporting the bot for spam, and can themselves be targeted by the defense. The defense has never sent any spam, and can report legitimate users to try to get their account suspended.

The Programmer's Paradox

I wrote a guest post today at The Programmer's Paradox, on open sourcing of personal coding projects.

Paul Homer is a great observer on software and engineering topics, I highly recommend subscribing to his blog.

Android Multicore

I have it on good authority that an Android release in late 2011 will support 29 CPUs. Here is how I expect the Q&A after the announcement to go, fielding questions from developers and users. In the interest of brevity I've omitted the questions. Here are the answers.

"No, not 'up to'. We support twenty nine cores."

"We are confident that ARM vendors are up to the challenge."

"Have you ever tried to animate a convincing look of horror on a green pig face? Plus, consider this: realistically dented helmets deformed by the physics, not pre-dented 3d models."

"No, that would be too many. Also, it needs to be a prime number to properly balance the workload."

"This is Google, of course we have data to support that."

"Very sophisticated CPU throttling, thats how."

 

I think I might stop correcting rumors. It's more fun to read the things people just make up.8 Feb 2011 via Twitter for AndroidDan Morrill
morrildl

On Self Driving Cars

Were I working on software for self-driving cars, I'd name the steering wheel device as /dev/student. That way the car would be controlled by the student driver.

Listing Processes with libproc

I recently had to work on functionality to look through /proc/<pid> for information about processes, which would have entailed an annoying amount of file schlepping and string parsing. Fortunately there is procps, a very nice library to makes /proc access work very much like directory access via opendir(). It normalizes the procfs implementations of a number of OSes like Linux and Solaris, so you work with a common data structure and don't have to maintain a bunch of parsing code. However it doesn't abstract /proc very much: you still need to know what it is and what information is in each /proc file to make good use of the facility.

You start with a call to openproc(), which creates a PROCTAB* structure to iterate through running processes.

#include <proc/readproc.h>

int main(int argc, char** argv) {
  PROCTAB* proc = openproc(PROC_FILLMEM | PROC_FILLSTAT | PROC_FILLSTATUS);

A flag argument to openproc() tell it what kind of information you want. The library will skip processing files in /proc if it can.

PROC_FILLMEM	read /proc/<pid>/statm
PROC_FILLCOM	allocate and populate `cmdline'
PROC_FILLENV	allocate and populate `environ'
PROC_FILLUSR	look up user id number, fill in user name
PROC_FILLGRP	look up group id number, fill in group name
PROC_FILLSTATUS	read /proc/<pid>/status
PROC_FILLSTAT	read /proc/<pid>/status
PROC_FILLWCHAN	read function name from /proc/<pid>/statm
PROC_FILLARG	handled identically to PROC_FILLCOM

The PROCTAB is repeatedly passed to readproc(), which populates a proc_t for each running process.

proc_t proc_info;
memset(&proc_info, 0, sizeof(proc_info));
while (readproc(proc, &proc_info) != NULL) {
  printf("%20s:\t%5ld\t%5lld\t%5lld\n",
         proc_info.cmd, proc_info.resident,
         proc_info.utime, proc_info.stime);
}

When done, call closeproc() to release resources.

closeproc(proc);

Some sample output from my system:

  process:  pages utime   stime
   xinetd:    139       1       0
     sshd:    866      10      21
     bash:   1377      28      16
ssh-agent:    208      11       3
  portmap:    158       1       4
rpc.statd:    208       1       0

 
proc_t

The proc_t contains a great deal of information about the process.

typedef struct proc_t {
  int
    tid,         // (special)     task id, the POSIX thread ID (see also: tgid)
    ppid;        // stat,status   pid of parent process

  unsigned long long
    utime,       // stat          user-mode CPU time accumulated by process
    stime,       // stat          kernel-mode CPU time accumulated by process
    cutime,      // stat          cumulative utime of process and reaped children
    cstime,      // stat          cumulative stime of process and reaped children
    start_time;  // stat          start time of process -- seconds since 1-1-70

  long
    priority,    // stat          kernel scheduling priority
    nice,        // stat          standard unix nice level of process
    rss,         // stat          resident set size from /proc/#/stat (pages)
  ...etc...

The proc_t also contains this maddening little member variable:

  unsigned pcpu; // stat          %CPU usage (is not filled in by readproc)

Instantaneous CPU percentage is commonly desired, but is not tracked by the kernel and is therefore not available anywhere procps can read. Tracking a percentage has to be implemented in the application by taking a snapshot, waiting a little while, and taking another snapshot to learn the utime+stime spent during the interval. This is the reason why top shows all CPU percentages as 0.0% when it starts, and corrects them on the next interval. procps provides a convenient place to store the CPU percentage, but does not implement it in the library.

Updating the Zodiac

Noting the impact of date corrections in stoking interest, Astrologers have announced further changes to the Zodiac for 2012. Instead of just 12 signs roughly corresponding to a month each there will be 8,766 signs, one for each hour of the year. The Hubble space telescope has provided ample stellar data to make this possible. Additionally to keep the Zodiac fresh, periodically old signs will be retired and new ones introduced.

Companies wishing to sponsor their logo as a new Zodiacal symbol should contact the International Union of Astrology and Tarot. One, five, and twenty year plans are available.

C++ POD Member Handling

I always mess up the initialization of plain old data fields in C++. Always. Maybe by writing it up, I'll finally get it right.

Plain Old Data is essentially anything a regular C compiler could compile. That is:

• integer and floating point numbers (including bool, though it isn't in C)
• enums
• pointers, including pointers to objects and pointers to functions
• some aggregate data structures (structs, unions, and classes)

A struct, union, or class is treated as plain old data if it has only the default constructor and destructor, has no protected or private member variables, does not inherit from a base class, and has no virtual functions. I suspect most C++ programmers have an intuitive feel for when a class behaves like an object and when its just a collection of data. That intuition is pretty good in this case.

The default constructor for plain old data leaves it uninitialized. An explicit constructor sets it to zero.

Code	Result
class Foo { public: Foo() {} int a_; };	Result: a_ is uninitialized.
class Foo { public: Foo() : a_() {} int a_; };	Result: the member corresponding to a_ is zeroed. Were it a structure, the entire thing would be zero.

People are often confused by the first point, that member POD fields are left uninitialized unless specifically listed in the initializer list. This is not the same as for member objects, which call the default constructor. Making this even more confusing, when a process starts up any pages it gets from the OS will be zeroed out to prevent information leakage. So if you look at the first few objects allocated, there is a better than average chance that all of the member variables will be zero. Unfortunately once the process has run for a while and dirtied some of its own pages, it will start getting objects where the POD variables contain junk.

 
Struct initializers

Putting a POD struct in the initializer list results in zeroing the struct. If the struct needs to contain non-zero data, C++0x adds a useful capability:

struct bar {
  int y;
  int z;
};

class Foo {
 public:
  Foo() : b_({1, 2}) {}
  struct bar b_;
};

Recent versions of gcc implement this handling, though a warning will be issued unless the -std=c++0x or -std=gnu++0x command line flag is given.

Automotive Easter Eggs

Automotive software has some great opportunities for easter eggs.

↑ ↑ ↓ ↓ ← → ← → OFF ON CAN	Richen fuel mixture 20%
VOL ↑ VOL ↑ VOL ↑ VOL ↓ HAZARD	Tune audio to Sirius headbanger channel
↑ ↑ ↑ ↑ HORN	Frickin' lasers!

IPv4 Addresses Almost Depleted

According to both ipv4depletion.com and potaroo.net, the IANA is within days of allocating its last block of IPv4 addresses to the regional internet registries. In fact it might have already happened, though there has been no announcement of it yet.

This means the global pool of IPv4 addresses has been exhausted, and all remaining IPv4 addresses have been allocated to a particular region of the planet. The regional authorities will start to run out in 2011 and 2012.

IPv6. It's on.

Requiem for Pluto

So long, Pluto. Gone, but not forgotten.