ZFS Spooky Failure at a Distance

I use Proxmox with a ZFS array to run a number of self-hosted services. I have been working on setting up zrepl for offsite backup, replicating encrypted ZFS datasets which the remote system will not be able to decrypt, only store.

 

While working through all of this, the new 28TB disk intended for the remote system appears to have failed.

root@zfsremote:~# zpool status
  pool: pool1
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
config:

        NAME        STATE     READ WRITE CKSUM
        pool1       DEGRADED     0     0     0
          sdb       DEGRADED     0    35     0  too many errors

 

Indeed, there are kernel messages about disk errors:

Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368396833 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368399137 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368397089 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368401697 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368401441 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368399393 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368402721 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368402465 ...
Mar 31 07:16:33 zfsremote kernel: I/O error, dev sdb, sector 23368402209 ...
Mar 31 07:16:34 zfsremote kernel: I/O error, dev sdb, sector 23368401953 ...

 

It seems odd, though. I had run `badblocks` destructive tests for weeks before moving on to creating the ZFS pool. After all that, it would choose this moment to begin uncorrectable failure?

Quite suspiciously, 07:16:33 is also the very instant when I sent a kill signal to a vzdump process running on the Proxmox host.

116: 2025-03-31 07:14:31 INFO:  29% (7.4 TiB of 25.5 TiB) in 9h 37m 2s
116: 2025-03-31 07:16:33 ERROR: interrupted by signal
116: 2025-03-31 07:16:33 INFO: aborting backup job

As I now know, trying to kill vzdump with a signal is not the right thing to do. `vzdump -stop` is the right way to interrupt it.

The OpenZFS docs say: "the following cases will all produce errors that do not indicate potential device failure: 1) A network attached device lost connectivity but has now recovered"

So far as I can tell, this is the explanation for this failure. Me sending a signal to vzdump interrupted the stream of ZFS operations, which manifested as a failed array on the other end. I've cleared the failure using `zpool clear` and will hope that zrepl will sort out bringing the two ZFS filesystems back into sync.

I plan to give it a day, then restore the remote dataset and check whether the file contents are sensible. The remote system does not, and will never, have the encryption key to be able to check the contents of the datasets it holds. I'll have to transfer them back to be able to access them.

Stadtarchiv Hannover bis 2026 geschlossen

I received a Sterbeurkunde from Stadtarchiv Hannover on 28 March 2025, with the following note in the email signature:

Von März bis Jahrsende 2025 verlagert das Stadtarchiv seinen Standort in das neue Sammlungszentrum an der Vahrenwalder Straße 321. Der Lesesaal ist geschlossen, die Bearbeitung von Anfragen eingestellt.

Bei der Erreichbarkeit unserer Kolleg*innen und unseres Funktionspostfachs stadtarchiv@hannover-stadt.de kann es zeitweilig zu Verzögerungen kommen. Wir bitten um Verständnis und freuen uns, Ihnen voraussichtlich ab Jahresbeginn 2026 am neuen Standort wieder im vollen Umfang zur Verfügung zu stehen.

Bitte beachten Sie hierzu auch die Informationen auf unserer Homepage unter www.stadtarchiv-hannover.de.

---

From March until the end of 2025, the city archive will relocate to the new collection center at 231 Vahrenwalder Straße. The reading room is closed and the processing of inquiries is suspended.

There may be temporary delays in reaching our colleagues and our functional mailbox stadtarchiv@hannover-stadt.de. We ask for your understanding and look forward to being fully available to you again at the new location from the beginning of 2026.

Please also refer to the information on our homepage at www.stadtarchiv-hannover.de.

In 7/2023 a request to Stadtarchiv Hannover would usually be answered in a week, but then something happened. Since 2024 response times have been 6-8 weeks. A post on their website mentioned a challenging staffing situation. I'm hopeful that in the long term, moving to a larger facility will help.

Imagery from the indexes was added to Arcinsys last year, those should still be available in the interim.

Macbook Air M1 USB-C Port Replacement

My Macbook Air M1 was gradually developing some kind of impairment in its USB-C ports where I'd have to jiggle or put actual pressure on a USB-C cable to get it to be recognized — and since it has no Magsafe port for charging, this meant it would switch to and from battery as its charging cable periodically lost contact.

Searching turned up people reporting similar issues, especially that the rear port started having a problem first until eventually the front port did as well. There wasn't a consensus solution but a replacement USB-C board from iFixit came up several times. For only $20, I ordered one.

The original USB-C board is off to the right side in this picture. One can see some corrosion and dirt, and also a bit of blackening on what I assume is a power pin. I believe that carbon buildup is likely the primary issue. I'll scrub it off with some alcohol on a cloth and put it away for the future, it would probably work again if needed.

Venmo Public Transactions

Venmo pushes hard for transaction activity to be Public. It doesn't say whether any past payments were actually public, and puts up an interstitial to confirm a change to Private.

This selection does have a benefit for the user, in making it more straightforward for friends to find each other and to make payment arrangements. However the choice has a larger impact on Venmo's user growth, and does come with downsides for their users like making activities public which they assumed were not.

Presumably Venmo has data on how much of a network effect they get from having payment information be Public, drawing in friends and family and acquaintances and randos. Venmo appears to allow this data to impact their UI design to steer users toward the choice most beneficial to the company.

EFF Privacy Badger

I started using Privacy Badger from the Electronic Frontier Foundation several months ago. It is a browser extension — I use it with Chrome — which blocks or restricts domains known to track identities and activity across the web.

One can click on the Privacy Badger extension icon to see what has been blocked, and also to make exceptions for the website being visited if needed.

 
 

This includes live links to tweets and other social media, which Twitter uses to gather data about the viewer. I allow these on certain sites which curate related tweets into stories.

I don't actively use Twitter any more but still find the zeitgeist there to be informative.

Ringing Endorsement for Signal

As published in The Atlantic today:

The Trump Administration Accidentally Texted Me Its War Plans
U.S. national-security leaders included me in a group chat about upcoming military strikes in Yemen. I didn’t think it could be real. Then the bombs started falling.
 
By Jeffrey Goldberg

The aforementioned group chat was using Signal. I guess that is quite the ringing endorsement of Signal's security and trustworthiness.

Signal was already being targeted by every nation-state and major criminal hacking group, I doubt that the knowledge it is being used for US war planning will especially increase the pressure they are under. As a family, we use Signal to coordinate everything important to us.

sync; sanc; sunc

# sync; sync; sync
#

Pros: Works.
Cons: Boring.
 

---

# cat ~/.profile
alias sanc=sync
alias sunc=sync
# 
# 
# sync; sanc; sunc

Much better.

Giant Airplanes Flying Low at 3am

Why yes, a Boeing 747 flying low directly over our house at 3am does wake us up. Who could possibly have predicted it?

Deleting Pokémon GO data

My final Pokémon

I was an enormous fan of Pokémon GO for several years, zealously playing no matter where I was. It spurred me to do interesting things like visit parks in my area to which I had never been, to capture the Pokestop. I joined in legendary battles. I recall captuing MewTwo at a local park with about a dozen other players. I'd been a player of Niantic's earlier game Ingress, and thought of Pokémon GO as a newer, shinier take on the concept.

As happens, my interest in catching Pokémon waned and eventually stopped by the summer of 2019. We went to LEGOLand and there is exactly one Pokémon screenshot in my photos, in what was surely a target rich environment. Emailed entreaties from Niantic to come back started a couple months later.

I knew that location data was the main economic reason for the game's existence. I wasn't especially concerned about it at the time, I felt confident that my visits to parks and monuments and fountains wouldn't be something to be concerned about.

 

That was then, this is now. The world seems more threatening, and Niantic's announcement of the sale of its games and location data to Scopely, which is owned by the Saudi Arabian sovereign wealth fund, is enough to trigger my spidey sense.

One can request deletion of the account and associated data from within the app if still installed, but it is not necessary to reinstall if already gone. Niantic has a request form to delete a Pokémon GO account. If you don't remember your account name, search your Inbox for the pleading entreaties to come back to the game — it went on for years.

After submitting the deletion request, Niantic sent an email requiring that I reply with a code to confirm the deletion within 30 days. Right now I'm trusting that they will actually delete the data: one person's information is valueless, there isn't a reason to lie about doing so.

Ausland Urkunden, Berlin Standesamt 1

In 2023 my wife and our children became German citizens via a declaration process called Staatsangehörigkeit § 5. The official paperwork came in the form of a document, an Urkunde über den Erwerb der deutschen Staatsangehörigkeit durch Erklärung, which we used to apply for Reisepässe.

The Urkunde durch Erklärung is very important, and will be needed to renew the passports. It is possible to replace it if something happens like fire or theft, but it isn't very straightforward to obtain the replacement. We decided to additionally file for birth certificates in Germany. These would be straightforward to re-order in the future as needed, and would serve as proof of citizenship. While at it, we also registered our marriage.

A civil records office in Germany is called a Standesamt, and registrations of foreign births are handled by Standesamt 1 in Berlin. The Berlin Standesamt 1 is famously backed up in processing submissions, we were advised to expect 2-3 years to process our forms.

Happily though, it effectively only took 5 months.

 

I say "effectively" because we missed the email of invoices to pay the Standesamt, until the Consulate sent them again 3.5 months later. So overall it took 8.5 months, three and a half months of which was on us.

Jun 17, 2024	Submitted forms at San Francisco Consulate.
Aug 2, 2024	Consulate forwards invoices from Standesamt in Berlin, which we missed seeing.
Nov 18, 2024	Consulate re-sends the invoices from Standesamt in Berlin, we paid the next day.
Jan 17, 2025	Recording date listed on the certificates.
Feb 7, 2025	Consulate receives the certificates from Germany.
Feb 28, 2025	Certificates delivered to us.

If you decide to do something similar, be aware that it is an expensive undertaking. Registering four births and one marriage cost 334 US Dollars in fees to the Consulate and a total of 630 Euros to Berlin Standesamt I. Altogether, the fees came to about a thousand US Dollars. One is not required to register the births in Germany, only do so if you believe it will be worth it.

Exporting 23andme Data

Our whole family had submitted DNA kits to 23andme including our kids and even my mother, who has since passed away. Our earliest kits used 23andme's v3 chip, the more recent ones used v5. However in the latter half of 2024 there came concerning news about 23andme's financial cicumstances. Last September we decided to export all of our data and ask that 23andme delete it. We didn't want it all to be handed over to a new buyer whose motivations we would not know.

We requested export and had to wait a bit for an email saying the exported data was ready. It generally arrived within a day of asking. Altogether each person's downloaded data is about 400 Megabytes, 375 Megabytes of which is in a single file: "imputed_genotype_data_r6." One of the requested exports seemed to get lost, but was processed on the second request. It took a few days altogether to request and download everything, then ask for deletion.

Nonetheless despite being concerned about what might happen at 23and me, we actually do want to continue to look for DNA matches to discover cousins and relatives. Each year at roughly the time of the RootsTech conference, MyHeritage offers free upload and processing of DNA data exported from other companies like Ancestry or 23andme. We uploaded the exported 23andme DNA data, omitting the kids for now.

We have decided to trust MyHeritage with our data because of a clear commitment in their privacy policy: MYHERITAGE HAS NEVER SOLD OR LICENSED GENETIC DATA OR HEALTH DATA, AND WILL NEVER DO SO IN THE FUTURE.

We'll watch for news if that ever changes, but such an unambiguous statement gives us enough confidence to proceed.

German Mothers Prior to 1975

My spouse's mother was German and emigrated to the United States in 1958. Until 1975 German mothers did not pass on citizenship to children born in wedlock. When that policy changed there was a process where parents could declare the citizenship of their children born before 1975, but it ended in 1977 and we think her mother never even knew of it.

So my wife was not born a German citizen.

 

Staatsangehörigkeit § 14

As the result of a court case in 2019, the modern state of Germany decided that this gender discriminitory practice where fathers would pass on citizenship and mothers would not had been unconstitutional. In 2020 an existing discretionary naturalization process called Staatsangehörigkeit § 14 ("StAG 14") was extended with a Müttererlass or mother's redress provision which relaxed some of the requiremens for descendants of German mothers in this circumstance.

We spent much of 2020 gathering documents and conducting genealogical research to prove her mother's German citizenship, as we no longer had the passport. The packet of documentation reached 77 pages altogether: German birth and marriage records, evidence of ties to Germany, sworn translations of English US documents to German, proof of financial sufficiency and health care, and more.

We mailed it in December 2020, and waited. The queue to process applications was several years long.

Hannover Neues Rathaus

 

Staatsangehörigkeit § 5

In August of 2021 while our StAG 14 application sat in the queue, a new and much simpler process to address historic gender discrimination in citizenship practices was introduced called Staatsangehörigkeit § 5 ("StAG 5"). It is a declaration, one declares one's German citizenship and provides evidence to support it.

The requirements are much simpler and the application is vastly shorter: it does not require evidence of ties to Germany, nor finanical information, nor any of the rest. It only requires the documentation of descent from a German parent within the timeframe and circumstances addressed by StAG 5.

• In May 2023 our original StAG 14 packet reached the front of the queue for processing. We received a letter from Germany noting the subsequent creation of StAG 5 and providing guidance if we chose to switch to it. We filled out the new forms to send in, relying on the documentation from the original StAG 14 submission for the rest of the supporting evidence.
• In September 2023 we received a letter describing several mistakes we'd made along with details of how to correct them, and one last bit of evidence needed. We submitted a response within a week.
• In December 2023 we received notice that the Declaration had been accepted. My spouse and our children were now dual citizens of Germany and of the United States. It was effective as of the date the declaration was made, in June 2023 when the StAG 5 forms arrived in Germany.
• We made appointments in February 2024 for each of them to apply for German passports, their Reisepässe.

If you are in a similar circumstance or a descendant, born to a German mother prior to 1975, the StAG 5 process is straightforward and can be done on your own. The cost is minimal, the application is free and obtaining needed documentation will generally cost less than one hundred Euros.

 

Getting Help

There are several available avenues for help:

• Reddit's /r/GermanCitizenship subreddit is amazingly helpful in advice and instructions. It unfortunately didn't exist when we started this process, and I only discovered it after we'd completed the entire thing. If I'd found it earlier it probably would have avoided the mistakes we had to correct in September 2023.
• It is absolutely possible to obtain documents from Germany on your own, and I wrote two blog posts about our experiences in doing so: #1 and #2.
• There are professional genealogists based in Germany active on /r/GermanCitizenship and available for hire, who can help obtain documents.
• One can hire a legal firm, though I won't link to any because I think that is a poor use of money. Legal firms won't help you gather the documentation you need, they will at most hire a genealogist on your behalf and then substantially mark up the fee. Hiring a lawyer is the most expensive way imaginable to fill out the StAG 5 form.

Google Photos Takeout Download using Firefox

Four months ago I exported all photos from Google Photos to import into immich, and scheduled followon Takeout runs every two months. I had naively assumed the subsequent exports would be incremental changes from the first one, but they are instead complete exports again. Two months later, download of the first scheduled Takeout repeatedly failed until Google disabled the download links, leaving me unable to download my photos at all.

After another two months, I made another attempt with the next scheduled export. I tried downloading the first archive using Chrome, and it failed twice. I switched to Firefox to download instead of Chrome, and it worked much better. Firefox appears to not give up so quickly and keeps trying. I was able to download all fifteen ZIP archives, fifty Gigabytes each, with only one download failure where I had to start it again.

Hurray!

Electrified USPS Fleet

In its initial proposal for a refreshed fleet of delivery trucks in 2022, the US Postal Service and its contractor Oshkosh Defense proposal listed a combined vehicle + cargo weight of 8501 pounds. This is a remarkably specific number, exactly one pound heavier than emissions rules would constrain. Were the vehicle 1 pound lighter it would be required to be considerably cleaner — and likely electric, to meet those requirements.

With more effort, and additional $3 billion in funding from the Biden administration, by the following year this proposal was revised and proposed that most of the fleet be electric. A gasoline version would be used for long routes in rural routes, areas without sufficient charging infrastructure. Initial electric vehicles were delivered in September of 2024, apparently to excellent reviews by postal workers using them.

Oshkosh has delivered only 100 electric trucks thus far, production was expected to ramp up from the initial deliveries a few months ago. The Oshkosh Defense CEO now says they'd be happy deliver the rest of the order as solely gasoline vehicles. It is difficult to see this in any positive way.

Google Cloud Four Cent Bill

I'd say not to spend it all in one place but I think the maximum number of places they could possibly spend it is four, so... go ahead. Live your best life, Google, with my compliments.