Avoid overclocking unless you use active cooling, you'll likely just make it overheat which will get you even worse performance and kill your pi over time. N64 optimization is definitely a rabbit hole so be prepared to experiment a lot. Try different emulator cores too if they're available. It'll look blurry as shit but it will run at a better frame rate. Lakka is a lightweight Linux distribution based on RetroArch that transforms a small computer like a Raspberry Pi into a full blown retrogaming console. See if you can find the menu for the emulator settings and drop the resolution down to 320x240, then work your way back up from there, that should make the biggest impact. But there's a lot more to it than just that. Flirc case for rpi4 is probably your best option if you want to maintain portability and not bother with fans. Overheating will cause the cpu (and gpu?) to be throttled to cool down a bit and cripple performance. The rpi4 gets way too hot without a really good heatsink or active cooling (fan or water-cooling). Easiest thing you can do is have good cooling. Mario Kart 64 runs pretty well so I usually test with that. The roms won't make a difference, though some games are doomed to run like shit no matter what.

Best n64 emulator lakka