WSL tricks part 2: VPN and MTU Setting

The first time you try to ssh to a remote server from WSL over a VPN connection, there's a good chance the connection will hang as soon as you start using it. After a little testing you may notice that it happens when using any command with longer output.

It turns out that the interface for the HyperV VM used to run wsl2 doesn't set its MTU correctly, and the network packets get discarded somewhere along the way. To fix this (assuming we don't have control over the VPN configurations) we need to set the MTU manually.

To find out what MTU setting should be used, the "tracepath" command can be used. Just install the package if needed, run "tracepath <server reachable by VPN>" and check the lowest "pmtu" value shown, that should be set as interface MTU.

For the configuration itself, a quick search on the internet did wield a few alternatives, but the nicer ones unfortunately didn't work... Setting the MTU directly on the HyperV configuration didn't get persisted across reboots; WSL's ubuntu doesn't seem to read any interface configuration on startup by default (and I don't see any good reason to install a init system on WSL)... 

In the end I've resigned myself and adopted the ugly solution: put the command to set it directly on ~/.bashrc:

#MTU
sudo ip link set eth0 mtu 1350

To get it at least a little less ugly, we can make it so sudo doesn't ask for a password: run "sudo visudo -f /etc/sudoers.d/mtu" and create the file with the following content:

ALL    ALL = (root) NOPASSWD: /sbin/ip link set eth0 mtu 1350

That will fix the hanging SSH problem. It's ugly because the MTU should be set dynamically / automatically and It's ugly because bashrc isn't the proper place for this kind of configuration, but at least it works.

WSL tricks part 1: Windows' openssh key handling

For a long time I've been using a very simple setup on Windows to manage Linux servers: just plain old putty, pageant, winscp and the like. With the recent implementation and advances on WSL, I felt it was time to test using WSL to have a better environment, and it has been a very good experience so far.

As this setup required a few tinkering around to get everything into place, I'll be writing up a few of the thoughts and tricks I've gathered along the way. For start, let's talk a little about something which caught me by surprise: the native Windows' OpenSSH ssh-agent key handling.

The first piece of our small puzzle to make a Windows station a good administration platform for Linux servers is to have ssh connection to those servers. Windows terminal is a good start, being a tabbed multi-terminal with support for cmd, powershell and WSL.

After that, it was a very interesting to discover there's a native port of OpenSSH for Windows available. At first sight, this could be an interesting tool to get this level of access without even needing to use WSL. But there's a catch...

To use ssh, you'll be willing to load your ssh key over an agent so you don't have to type your super-secret-long-passphrase every time, having it work automagically during your work session. The concept around ssh-agent is that your decryptographed key will stay in memory, never touching disk / permanent storage, and as such will be securely erased when you unload it and/or power down (assuming no one is going to do some crazy ninja cold RAM data extraction to get your ssh-key).

By the other hand, Windows' openssh port just disregarded the whole concept and instead permanently store your ssh key on Windows Registry, which in turn get persisted on disk itself. While it seems to be cryptographed over some kind of user-context security, it didn't really strike me as particularly safe, and there's some article somewhere explaining how it could be recovered by someone with access and how it would be very hard to get it fully out of disk once it gets there.

While I didn't dig much deeper in this, what I saw was enough to convince me to stay away from windows' openssh - there certainly seems to be some risk, and wsl + ubuntu ssh works perfectly fine for my needs. 

Incremental pvmove

I've recently had some fun with a BTRFS filesystem corruption (described in a gist here). Ended up with the recovered data in a linear LV formed by 2 non-redundant PVs.

After all needed checks were done and a fresh backup taken, it was time to eliminate the failed btrfs filesystem and get the data back on redundancy by reusing its RAID5 array. In my mind, it would be simple a question of pvmove'ing the data, which I could do over time, no hush, by making partial/incremental data moves.

Contrary to what I expected however, it turned out I couldn't just run pvmove for some time, cancel it and have the data which was already moved stay on the new volume; instead, after stopping pvmove the lvm structure stayed as it was at the start.

It turns out pvmove basis for data movement is the segment, that is, contiguous allocations of data, not individual extents as I expected. For the data movement to be "commited" it would have to finish moving the whole segment, which in this case would mean moving the entire PV / disk partition, as it's fully allocated to the same LV.

As pvmove also supports specifying specific areas of the PV to move, It seemed right to script my way to the result I wanted, so I've made a small script to sequentially pvmove a disk in "small" [parameterized] amount of extents per time. The script can be interrupted and the fully finished pvmove's stay as they should, so you can get back another time and continue from (mostly) where you stopped.

Please keep in mind that this has been made and tested in a very specific case - PV allocated sequentially to a single LV. I didn't really try to cover other user cases, so it could move data it shouldn't, and while I don't expect a pvmove to cause data loss, if the script somehow gets the start extent or size wrong, I can see it getting the LV / filesystem fragmented. As usual, having a backup ready is recommended.

Without further ado, my hackish incrementa_pvmove.sh script is available here: 

https://gist.github.com/grmontesino/8ec29cd16cf3d893dde808f35f079304

Hardware failures...

The Bad:

When I got to work today, I've found our server (white box with Debian etch responsible for networking, files, printers, etc) powered off - most likely the UPS' battery didn't survive a power outage during the night. When I turned the server on I was greeted by all those nice lines telling me I had a hard disk problem.

The Ugly:

Instead off marking the disk showing the read errors as bad, the RAID stack (device mapper?) somehow concluded the "good" disk of the RAID 1 array wasn't synched and kicked it out...

The good

The bad sectors did take only some unimportant collectd status files with them. After some poking with dd trying to force the HD to redirect the bad sectors, the read errors vanished and the Reallocated Sector Count didn't increase according to smartctl, which seems like a good signal.

Moving and resizing partitions

Earlier today I've migrated all the data from a 80 GB HD to a new one of 250 GB. It was a pleasing surprise to see how far the support for these kinds of things have gone.

The old HD had 4 partitions: The first two for Windows XP on NTFS (not my computer...), the third a small linux swap and the last a ext3 root partition for ubuntu.

The process was quite simple: Set the jumper on the new (samsung) HD, as the motherborad has one of the VIA chipsets without support for the 3Gbps models, connect both of them, boot with an ubuntu live CD, dd everything from the old HD to the new one, disconnect the old HD, boot with the live cd again and finally expand the partitions on the new HD using gparted.

The big surprise came in the last step above: I really didn't expect resizing NTFS partitions through Linux to work, and it went flawless (actually, I had a small glitch on resizing the ext3 filesystem, which forced to run resize2fs by hand with the force [-f] flag, as it kept asking for fsck).

More RAM 2

Note to self: Whenever adding RAM, besides looking for the correct count on POST, it seems to be a good idea to see also on kernel's boot logs if the kernel found and can use all the available memory.

In this case, I had to recompile the kernel with HIGHMEM enabled. Well, the kernel I had running (2.6.18) was starting to get old anyway, so now I've got 2.6.24.7, which seems to be the latest one on which the nvidia binary drivers will compile.