This reveals the core truth of things: “People do model merges because it’s easy. Just a few clicks of a button and you can release a new model. Training new models is hard, and so very few people do it; the tragedy of open source.”
When there’s an end goal in sight, it’s easy for people to throw a bunch of CPU time at merging models and seeing what happens. It’s the opposite problem of top research labs, where there are very few core people who need tons of resources to make real progress. On the internet, there are millions of people who will happily press go and see if they get pretty pictures out
Jun. 2023: Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards This paper shows that instead of needing to train many models on linear scaling of reward functions with a multi-objective RL approach, one can simply fine-tune multiple models with respect to each reward of the rewards and merge them. The actual reward functions are quite buried in the Appendix (don’t do this for core details!), but they tend to be quality evaluators by different reward models.
