This process is not done through the collection, but rather through the settings of the video you want to replace. To learn how to replace a standalone video, please read this article. Doing this will quickly swap the video out and all collections that referenced the old video will now reference the new one.
If you have already uploaded the new video normally without using the ‘Replace Video’ feature, you will want to add the new video to the collection while keeping the old one in it. You can then go into the settings of each button that references the old video and select "Change Video". The currently referenced video will be selected by default. Simply select the new video and click "Save". Once all buttons or scenes have been updated, the old video can be removed from the collection.