Wednesday, January 24, 2018

I Know What You Are by the Smell of Your Wi-Fi

In July 2017 gave a talk at DEFCON 25 describing a technique to identify the type of Wi-Fi client connecting to an Access Point. It can be quite specific: it can distinguish an iPhone 5 from an iPhone 5s, a Samsung Galaxy S7 from an S8, etc. Classically in security literature this type of mechanism would have been called "fingerprinting," but in modern usage that term has evolved to mean identification of a specific individual user. Because this mechanism identifies the species of the device, not the specific individual, we refer to it as Wi-Fi Taxonomy.

The mechanism works by examining Wi-Fi management frames, called MLME frames. It extracts the options present in the client's packets into a signature string, which is quite distinctive to the combination of the Wi-Fi chipset, device driver, and client OS.

The video of the talk has been posted by DEF CON:


  • The slides are available in PDF format from the DEFCON media server, and the speaker notes on the slides contain the complete talk.
  • The database of signatures to identify devices is available as open source code with an Apache license as a GitHub repository.
  • There is also a paper which describes the mechanism, and which goes a level of detail deeper into how it works. It is available from arXiv.

Tuesday, January 23, 2018


Behold: the Yakthulhu. It is a tiny Cthulhu made from the hair of shaved yaks.

(Really, it is. Yak hair yarn is a thing which one can buy. Disappointingly though, they do not shave the yaks. They comb the yaks.)

Saturday, October 21, 2017

On CPE Release Processes

Datacenter software is deployed frequently. Push daily! Push hourly! Push on green whenever the tests pass! This works even at extremely large scale, new versions of are deployed multiple times each day (much of the site functionality is packaged in a single deployable unit).

CPE device software tends to not be deployed so often, not even close. There are several reasons for this:

  • Test practices are different.

    Embedded systems is one of the oldest niches in software development and does not have a strong tradition even of unit testing, let alone the level of automated testing which makes concepts like push-on-green possible. One can definitely get good unit test coverage of code which the team developed, but the typical system will include a much larger amount of open source code which rarely has unit tests and is daunting for the team to try to add tests to. Much of the code in the system is only going to be tested at the system level. With effort and effective incentives one can develop a level of automated system test coverage... but it still won’t be close to 95%. System level testing never is, the combinatorial complexity is too high.

    Additionally, with datacenter software, the build system creating the release is often somewhat similar to the production system which will run the release. It may even be the same, if the development team uses prod systems to farm out builds. A reasonable fraction of the system functionality can be run in tests on the builder.

    With CPE devices, the build system is almost always not a CPE being tasked to compile everything. The build system is an x86 server with a cross-compiler. The build system will likely lack much of the hardware which is key to the CPE device functionality, like network interfaces or DRM keystores or video decoders. Large portions of the system may not be testable on the builder.

  • The scale is different.

    Having a million servers in datacenters is a lot, that is one or more very large computing facilities capable of serving hundreds of millions of customers.

    Having a million CPE devices is not a lot. There are typically multiple devices within the home (modem, router, maybe some set top boxes), so that is a couple hundred thousand customers.

    It can simply take longer to push that amount of software to the much larger number of systems whose network connections will generally be slower than those within the datacenter. Multiple days is typical.

  • The impact of a problem in deployment is different.

    If you have a serious latent bug which is noticed at the 3% point of a rollout within a datacenter, that is probably a survivable event. Customers may be impacted and notice, but you can generally quarantine those 3% of servers from further traffic to end the problem. The servers can be rolled back and restored to service later, even if remediation steps are required, without further impacting customers.

    If you have serious latent bug which is noticed at the 3% point of a rollout within a CPE Fleet, you now have a crisis. 3% of the customer base is impacted by a serious bug, and will feel the impact until you finish all of the remediation steps.

    If the remediation steps in 3% of a datacenter rollout require manual intervention, that will be a significant cost. If the remediation steps in 3% of a CPE Fleet deployment require manual intervention, it will have a material impact on the business.

We’ll jump straight to the punchline: How often should one deploy software updates to a CPE fleet?

In my opinion: exactly as often as it takes to not feel terrified at the prospect of the next release, no more and no less often than that.

  • Releasing infrequently allows requirements and new development to build up, making the release heavier and with more opportunities for accidental breakage. It also results in developer displeasure at having to wait so long for their work to make it to customers, and corresponding rush to get not-quite-baked features in to avoid missing the release.
  • Releasing too frequently can leave not enough time to fully test a release. Though frequent releases have the advantage of having a much smaller set of changes in each, there does still need to be a reasonable confidence in testing.

In the last CPE fleet I was involved in, we tried a number of different cadences: every 6 weeks, then weekly, then quarterly. I believe the 6 week cadence worked best. The weekly cadence resulted in a number of bugs being pushed to the fleet and subsequent rollbacks simply due to the lack of time to test. The quarterly cadence led to developers engaging in bad behavior to avoid missing a release train, by submitting their feature even in terrible shape. The release cadence became even slower, and the quality of the product noticeably lower. I think six weeks was a reasonable compromise, and left enough headroom to do minor releases at the halfway point as needed where a very small number of changes which were already tested for the next release could be delivered to customers early.

One other bit of advice: no matter what the release cadence is, once it has been going on long enough, developers will begin griping about it and the leadership may begin to question it (Maxim #4). Leadership interference is what led to the widely varying release processes in the last CPE fleet I was involved in. My only advice there is to manage upwards: announce every release, and copy your management, to keep it fresh in their minds that the process works and delivers updates regularly.

Wednesday, October 4, 2017

Bad Physics Jokes

Recently I posted a number of truly terrible physics jokes to Twitter, as one does. For your edification and bemusement, here they are:

  • If you accelerate toward the red light fast enough, the blue shift turns it green again.
  • Two people are walking up a frictionless hill...
    (that's it. That's the joke.)
  • Whenever a neutron asks the price of anything: "For you, no charge."
  • Whenever the proton is asked if they are sure: "I'm absolutely positive."
  • The electron isn't invited to the party in the nucleus. The other particles find it boorish, being so constantly negative all the time.
  • The Higgs Boson conveys the gravity of the situation to other particles.
  • Photons make light of EVERYTHING. They can be SO inappropriate sometimes.
  • You might assume that Gravitons would be extroverts attracted to large groups, but no. They're actually really, really shy.
  • Despite the occasional unverified sighting, experts agree that Phlogistons are the Bigfoot of the subatomic particle world.
  • Neutrinos are the tragic poets of the subatomic world. They yearn for interaction, but know that it can never be.
  • As a community, Protons realized that their diet and exercise habits needed to improve.
  • Other particles wish they could help the Pion, but don't know what to do. Even the smallest thing can make them fall to pieces.

Sunday, September 24, 2017

There is No Feminist Cabal

From the 23-Sep-2017 New York Times:

One of those who said there had been a change is James Altizer, an engineer at the chip maker Nvidia. Mr. Altizer, 52, said he had realized a few years ago that feminists in Silicon Valley had formed a cabal whose goal was to subjugate men. At the time, he said, he was one of the few with that view.

Now Mr. Altizer said he was less alone. "There’s quite a few people going through that in Silicon Valley right now," he said. "It’s exploding. It’s mostly young men, younger than me."

I want to share some experiences, as another white male in Tech for a similar number of years as Mr. Altizer.

A while ago I made a conscious effort to follow more women in Tech on Twitter, to deliberately maintain a ratio of ~50% in those I follow. I wanted to try for more perspective than that provided by my own vantage point in the industry, where the gender ratio is definitely not 50%. It has been illuminating... and often painful. Intermixed with happy and proud events in life and work is the constant level of sexism which women experience. Sometimes it is blatant and vile: intimidation, physical threats. More often it is a grinding, ever present disrespect from men. It is so commonplace that it becomes completely expected, often mentioned in an offhand way. This doesn’t mean it is a minor thing, it means that it never stops, isn’t possible to avoid, and ceases to be surprising.

You likely won’t hear this in person, from women you work with. That doesn’t mean women you work with are experiencing something different. It doesn’t mean that their career and work environment are free of sexism and discrimination. It means that talking about it in person is asking them to relive those events, sometimes extraordinarily painful events. It means it is vastly more difficult to relate horrible experiences in person, in conversation. It is understandable to not want to talk about it.

Yet one thing I haven’t seen, not even a hint of, is the existence of a powerful group of women who are organizing to oppress men. I’ve seen no evidence of any kind of backlash against men in Tech. Jokes about a Cabal started circulating after the NYT story was published. This was irony, not confirmation.

We’re hearing more about sexism in Tech, far more than we did even a year ago. I think, I hope, that is because we are in the early stages of the extinction burst. When a behavior which was formerly rewarded no longer is, that behavior will begin to decline... except for a final gasp, a final burst, in trying to turn back the clock. The process of acknowledging the disparities in Tech has been ongoing for many years, slowly. It has reached a point where the industry is starting to respond, if only a little. That the response may grow stronger will feel like a threat, like a backlash against males. It really isn't. It is about disparity, and doing something to rectify that long-present disparity.

I’m posting this because it is unfair to expect people in disadvantaged groups to carry the entire burden of correcting the disadvantage. In Tech the advantaged group is males, and as I happen to be a male in Tech, that means me. I have not been nearly enough a part of the solution. That needs to change.

Males in Tech have almost certainly witnessed aggressions: a woman being spoken over, or not being invited to a meeting she should be, or not receiving sufficient credit for her work. One thing learned from following women in tech is that no matter how much we think we understand, the aggressions they go through happen orders of magnitude more frequently than we think. For every occurrence we know of there are ten more, a hundred more, a thousand more, which we don’t see and don't grasp the frequency of because we are not female.

We, males in Tech, need to speak out. We need to speak out frequently and firmly. So I am. My voice isn’t powerful, but power can be achieved in numbers too. I’m adding my voice to the multitude. You should too; not just as a blog post, speak out when you see the grinding aggression happening.