So I had to find again what I did last time to resolve that.
And adapted what I found on this mailinglist (https://lists.fedoraproject.org/pipermail/mutt-kz/2012-April/000046.html):
notmuch search --output=files -- tag:archive and not tag:archived | xargs -I{} mv -f {} /home/$USER/Maildir/Archive/cur/
Then I used (almost instantaneous):
notmuch tag -archive +archived -- tag:archive
But something went wrong, and I got all my message in duplicate (with and without the archived tag), panicked, used rmlint (https://github.com/sahib/rmlint) to remove the duplicates, this removed the ones taged with archived of course (I have backups, I don’t care messing with things, don’t do that at home).
And I remembered, hey I can ask notmuch to search for all emails in a physical maildir folder. So, running that, (1 minute runtime):
notmuch tag +archived -- path:Archive/cur
But why stop here, why not moving everything to the archive, it took 5 minutes (for 80,000+ emails) including the notmuch new (which was the slowest part, took almost 4 minutes):
notmuch search --output=files -- not path:Archive/cur and not tag:inbox and not tag:unread and not tag:archive and not tag:archived | xargs -I{} mv -f {} /home/$USER/Maildir/Archive/cur/
Problem solved, took 15 minutes , but at least next time it will take me 2 seconds. Thanks again notmuch!