The elements of the view matrix were so tiny that its determinant was almost zero, thus the inverse couldn't be computed and it used the identity matrix instead. That's why the returned numbers were in the range [-1, 1].
I just replaced my fuzzy compare for the determinant with a strict compare (we just want to avoid divide by zero, so very tiny numbers can still be considered as valid).