alt Hacker NewsDeepSWE is the benchmark you want to actually look out for. Only one that aligns with actual user reported results from trying the models.
DeepSWE is the benchmark you want to actually look out for. Only one that aligns with actual user reported results from trying the models.
Did you read the blog post? They compare to deepswe and call it out as the worst one for false positives (failed, but the benchmark assessed it as correct). It also has less language variance.