For the other part of the issue (the truncation in SHA512), in Rust you could've had:
struct Input<'a>(&'a [u8]);
impl<'a> Input<'a> {
fn new(input: &'a [u8]) -> Result<Self, Error> {
if input.len() >= 256 {
return Err(Errro...);
}
Ok(Self(input))
}
}
Now the truncation is a terrible idea in the first place, but at least with this it would have been entirely obvious to the caller that it was there.
In C you could do just the same: https://uecker.codeberg.page/2026-10-24.html Whether you should is a different question.
In any case, you can not change the API of an existing function. But returning NULL for error has required by POSIX would have been just fine.