@6 don't force restart, repeatedly manually invoke the oom killer with sysrq+f until it unfreezes (assuming you have magic sysrq enabled). I have to do this all the time.
(and yes, linux's memory management sucks and the allocator has nothing to do with it. I wish overcommit was off by default but opt-in per allocation using MAP_NORESERVE, instead of MAP_NORESERVE having no effect if overcommit is disabled. also it should take swap into account, I didn't realize it didn't)