logoalt Hacker News

Topfi • last Tuesday at 11:26 PM • 1 reply • view on HN

Very task-dependent of course and mine are unique to say the least, so could see Luna being better in certain domains, even if my measurements have not shown that yet, happy for anyone to show otherwise.

For what it's worth, ran every task twice on each model, most were for some UI component synthesis and charting insanity that is a bit hard to explain, but some were simple tag selection, basic noul at threshold 60%. Essentially, whether to use the provided tag given the title of a browser tile:

  1. Title: "Mortgage calculator: estimate your monthly payment (Bankrate)"
     Tag:   "house hunting"
     Jev:   0.69 (yes), 0.72 (yes)
     Luna:  0.21 (no),  0.21 (no)

  2. Title: "S&P 500 index: live chart and news (Bloomberg)"
     Tag:   "investing"
     Jev:   0.91 (yes), 0.90 (yes)
     Luna:  0.56 (no),  0.56 (no)
Of course, tags can be a bit subjective, but in these cases, I'd argue the values provided by Jev were far more representative of my subjective assessment over Lunas. If SnP stuff on Bloomberg isn't investing, nothing is.

Goal for tagging is mainly a near instant, over writable, sane default provided to users in the background. Resolve the whole "I love using Notion/Obsidian/PKM software of your choice but spend 80% of my time just thinking about the ideal tag before starting to read" issue. Lunas output is not really helpful here.


Replies

brianyu8 • yesterday at 5:00 AM

I tried reproducing your examples as predicate questions on the Decisions API (https://gist.github.com/by-openai/7b376d7866e36a4f38581f73ce...) and got:

  1. Title: "Mortgage calculator: estimate your monthly payment (Bankrate)"
     Tag:   "house hunting"
     Decisions API (2 runs):  0.99, 0.99

  2. Title: "S&P 500 index: live chart and news (Bloomberg)"
     Tag:   "investing"
     Decisions API (2 runs):  1.0, 1.0

If you have other examples of requests with unexpected outputs, feel free to email me at [email protected] and we can try to get to the bottom of it. Thanks for trying out the API!
➕ show 1 reply