Monday, June 15, 2026

Electric Vehicles Shall Prevail

US policy toward electric vehicles abruptly changed in 2025, rapidly winding down the sales incentives which had previously been in place and resulting in blunting what had been healthy growth in the sales of electric vehicles. Rising gasoline prices in 2026 has restored some of that interest, but primarily in used electric vehicles now as supplies of new vehicles have shrunk.

Nonetheless the fact remains: sales of internal combustion vehicles peaked about a decade ago and have remained at best flat since, while electric vehicle sales are growing rapidly. Don't believe it? Sales peaked in the US in 2015, and 2017 world wide.

bar chart of US electric and non-electric vehicle sales by year, with non-EV peaking in 2015 bar chart of world-wide electric and non-electric vehicle sales by year, with non-EV peaking in 2017

China is the dominant market for electric vehicle sales, larger than the US and the European Union put together.

Sales of electric vehicles each year in the US, EU, and China. China sales diverge and accelerate rapidly after 2020.
 

In 2026 electric vehicles have achieved purchase price parity with gasoline vehicles, and with vastly lower costs per mile. US policy shifts will slow vehicle electrification in the US market, but globally it is accelerating.

EVs have won, all that is left is the circus.

Monday, June 8, 2026

Profile-Driven Observability

a graph of a metric showing a regular series of dips

Postmortem Culture means getting more actionable followups out of each Postmortem, and a very frequent action item we look for is, "where did we not have the visibility that we wanted to have?" Adding metrics, log messages, and trace points are all fine action items to take after an outage. Surely there is something which could have been more clear and reduced the time to remediate.

Observability vendors already provide help in instrumenting a codebase, such as Honeycomb's Agent skills and MCP service. I'd like learnings from a Postmortem to flow back into the tooling like a profile-driven optimization: take our handling of recent incidents into account when suggesting instrumentation. Even if the tooling might not otherwise suggest adding visibility in a particular area, if it is something we have struggled with then it is worth adding more help.

Monday, June 1, 2026

Observability Agents and Postmortem Culture

A service I wish to will into existance would help us better leverage our observability work, our traces and metrics and logs, in incident response during an outage.

  • I want the service to be part of a channel where we’re working on an outage, in Slack or Microsoft Teams or in some incident response tool, so the agent can see what is being worked on and get context of what we’re looking at.
  • I want the service to continue helping as we close the incident and proceed to followup on issues noted.
  • I want the service to guide us to do more extensive followup after an incident, not just the most proximate causes.

 

I want it to offer proactive help during the stressful work for the team, for example:

  1. Unprompted interjection of new information.
  2. Handle comms and quell panic.
  3. Guide us to declare the end of the Incident.
  4. Help draft the Postmortem.
  5. Make the Postmortem better.

 
1. Unprompted interjection of new information

I want the agent to call out outlier metrics/traces/etc which we don’t appear to already be aware of. It is important to set the threshold appropriately: repeatedly interrupting us for things we already know would be unwelcome, interjecting about something relevant which we don’t appear to have noticed could be a lifesaver. I think this part of the functionality is already being addressed, for example by Honeycomb Canvas. It is a natural progression for an observability tool.

2. Handle comms and quell panic

Someone needs to reflect pithy summaries of what is happening to a Slack or Teams channel for the rest of the company, to provide reassurance and quell panic. We write SOPs to emphasize Comms, but people in the thick of it get focused on the firefighting and sometimes neglect to do so. Summarizing the state of the response and adjusting its level of detail for a broader and less technical audience seems like something a suitable LLM could do.

3. Guide us to declare the end of the Incident

Sometimes a problem just trails off once we’ve addressed enough of its causes, but we may not decide to close the Incident until well after the point where keeping it open is really warranted. I'd like the service to let us know that things appear to be on the path to normalcy and estimate how long before the changes it is seeing would bring us back into the usual range.

4. Help draft the Postmortem

After the incident, the agent should help us write the Postmortem. Tooling often only focuses on the straightforward parts of that: the summarized description of the problem and a timeline, especially if it can annotate the timeline with graphs of impacted metrics and relevant details.

That would be dandy and would save us time, and help get the Postmortem out with less delay — which is important, to be sure. We want to let people give feedback and contribute to the Postmortem while everything is still fresh. I think this part of the functionality is already being addressed, for example incident.io

but also...

5. Make the Postmortem better

Vastly more valuable in the Postmortem would be to help us extract more actionable followups, not just the immediate triggers but as many things which could be better as we can find. If the agent can see negative trends in metrics or traces which do not appear to be a direct result of what we’ve identified as the root cause, being able to implement more fixes and improve the system's robustness without needing to suffer through another incident first would be very valuable.

Most of the incident-focused products available and work that I've seen focuses on the description and the timeline and supporting data in a Postmortem, which is dandy and saves time in the writing of it, but those are part of the Archaeology when what I really want to focus on is the Future. Even things we might not be positioned to take on for a while, we could still try to address proactively before they happen again.




Postmortem Culture

We do our best to design in redundancy and robustness and build reliable systems, but we always end up responding to failures which we didn't adequately control for and improve the reliability of the system over time as it operates. One of the primary tools to do this is the Postmortem, where we describe a problem which happened and list off what we are going to do about it.

A scene from the Simpsons with many rakes arrayed around the ground.

We want to learn as much from every incident as we can. We want to address as many weaknesses in our system as we can, without having an outage for every one of them. If we can identify more things which went wrong, things which were perhaps not the primary problem but nonetheless still a problem, we accelerate the process of improvement. Making maximal use of Postmortems to improve the system is Postmortem Culture.

Every outage starts with stepping on a rake and being hit in the face. We should be able to look past the rake which just hit us in the face, and look around for nearby rakes which we haven't stepped on yet. Postmortem Culture is the rakes we did not step on.




Existing Products

1. Honeycomb Canvas is an existing product in this space, particularly the live assistance during an incident using observability data.

Sample page from Honeycomb Canvas, showing chat boxes and graphs
 

2. incident.io is another product in this space, especially in helping to draft postmortems — the Archaeology part of the postmortem, at least. incident.io is evolving from an on-call and incident management tool, not an OpenTelemetry collector. The assistance it can currently provide during the incident is more in looking for patterns with prior incidents.

incident.io chat window where the agent answers questions about prior incidents with similar symptoms
 

3. There are a few products which describe themselves as a Virtual SRE team, though I don't really like that term. An experienced SRE team is a hugely valuable resource and the tools I've seen are at best automating a small part of what SRE would do. I'll write more about these kinds of products as I learn more about them.

Tuesday, May 26, 2026

Renewable power generation in Texas

Texas has long been a leader in the deployment of solar and wind, though conflicted about it. Power outages in Texas are often attributed to wind and solar, with the sage observation that the wind doesn't always blow and the sun doesn't always shine. With the Texas grid mostly isolated from the rest of the US, gaps in generation mostly cannot be ameliorated by importing power from elsewhere.

For example a few months after the February 2021 extended winter power outage in Texas, FERC reported that, "Natural gas-fired units represented 58 percent of all generating units experiencing unplanned outages, derates or failures to start. The remaining portion was comprised of wind (27 percent), coal (6 percent), solar (2 percent) and other generation types (7 percent), with four nuclear units making up less than 1 percent."

But during the outage the administration in Texas called out Wind as the primary culprit. Later corrections never achieve the same reach as the initial blame does.

Yet nonetheless, in 2026 Texas has reached several big milestones in its conversion to renewable power. This is amazing, commendable news.




Recent Milestone #1: Solar Surpasses Coal

In a report published on May 13, 2026, the US Energy Information Administration said, "In our most recent Short-Term Energy Outlook, we forecast that annual electric power generation from utility-scale solar will surpass that from coal for the first time in 2026 within the electricity grid that covers most of Texas."

Source of Texas power over time showing solar growing strongly but methane growing as well

So far, solar is not displacing coal so much as it is shutting out coal from new capacity being built. This is almost guaranteed at this point: not only does solar have no fuel cost, it is cheaper per Megawatt to build. It is difficult for coal to compete with a technology which is both less expensive to operate and less expensive to deploy.

Unfortunately though, as shown in the chart, electricity from methane (I decline to use the term natural gas) is also growing strongly.




Recent Milestone #2: Texas Battery Capacity Surpassed California

In the longer term, the most pivotal technology deployment isn't the megawatts of capacity of wind or solar generation, it is the megawatt-hours of battery storage deployed. Grid-scale batteries are how renewable power can address "the sun doesn't always shine." argument often advanced in favor of fossil generation. For example the Heritage Foundation in April 2025 called for policies to discourage deployment of renewables and battery storage in favor of fossil sources.

Grid connected battery capacity in Texas each year showing robust growth from 2020 through the beginning of 2026

At some point in the last several months, Texas passed California in total battery capacity deployed. This is a big milestone, and a wonderful accomplishment.

Some day soon I hope to write about renewable generation in Texas passing Methane. It is, at this point, inevitable.

Friday, April 10, 2026

Bundesarchiv militärischer Unterlagen

Military records of your German ancestors can be illuminating, though it is very hit-or-miss what records survive. You can order military records using the forms at the Bundesarchiv.

We sent a request to poststelle-pa@bundesarchiv.de in Deccember of 2025, and in April 2026 received an email response with PDF files of all records found. For our great-grandfather's service in WW I, only the index of two hospital stays in 1917 remain.

We later received the bill in the postal mail for a quarter hour charged at 20 Euros/hour, so 5 Euros total. They accept SEPA payments to a bank account number called an IBAN, which we used wise.com to pay for a few cents in fees. The Bundesarchiv also includes instructions of where to send cash Euro notes if you wish, though personally I wouldn't recommend doing that.

Monday, January 5, 2026

Twenty Eight Kilowatts

SolarEdge CCS charging cord

Our 2016 Kia Soul EV is able to charge at about 7 KWatts using our charger at home, a level2 EV charger built into the SolarEdge solar inverter. It has a CCS plug.


 
Larger CHAdeMO plug hooked to a white Kia Soul

Today I charged the car while out and about, using the other charging port on the Kia: CHAdeMO, a DC high power charging standard. I should put "high power" in scare quotes as it is quite pedestrian by today's standards, it is a standard from years ago.

  • CHAdeMO managed to deliver 28 KWatts to this car.
  • Twenty. Eight. Kilowatts.
  • Four times more power than the CCS charger at home.

It charged from 62% to 90% in 22 minutes, when it began to slow as the battery approached full. I unplugged it at 90% and continued on my merry way.

EVgo charging control screen showing 28 KWatts power delivered, 1.441 kWh energy dispensed in 3 minutes 14 seconds