I haven't been able to produce a short, verifiable example, but it is looking like this might be related to using SoundRecorder instances as members of a form (likely WinForms being a pain, I figure?). Adding the following to the form's closing event fixes the issue:
private void FFTWindow_FormClosing(object sender, FormClosingEventArgs e)
{
this.sampler.Stop();
}
As long as the processing thread is stopped directly as above, no trouble. Not entirely sure why this would happen, but I don't mind needing this workaround.