OoT and MM are my two favorite games of all times, so obviously they loom large when I look back on the N64 era. However, the console had plenty of awesome games even when disregarding those two masterpieces. Here is a round-up of my favorite non-Zelda N64 games:
Perfect Dark
Easily my favorite FPS, Perfect Dark blew Goldeneye out of the water and was in many ways far ahead of its time. In terms of gameplay it was like Goldeneye, only better and more polished, the multiplayer was one of the best ever, there were tons and tons of different modes, the replayability was almost limitless, the weapons were amazing, the graphics were stunning by N64-standards and the soundtrack was excellent as well.
Paper Mario
One of my last N64 games was also one of the best. It was such a blast of fresh air when it came out, started off great and just got better and better the more you played it, to the point where even the credit roll was brilliant (anyone who has finished Paper Mario will back me up on this). The game was just so much fun, so well designed, so genuinely funny and full of cool references to past Mario games that you just never wanted it to end. A wonderful experience, no doubt about it.
Jet Force Gemini
One of the more overlooked Rare classics for the N64, Jet Force Gemini is nonetheless an absolutely massive, epic space opera that will last you a long, long while. Not only was it totally huge, it was also great fun and a real challenge even for the hardcore gamer (Mizar was brutal!). It delivered in terms of variety too, and featured three unique playable characters, plenty of big, honking weapons and a surprisingly solid multiplayer-mode (this was when Rare started adding multiplayer-modes to what would normally be just singleplayer games). Not to mention, the graphics were fantastic for the time and the soundtrack was even better.
Banjo-Tooie
Another Rareware classic, Banjo Tooie is my favorite out of all their 3D platformers. It is basically Banjo-Kazooie on steroids, which I consider to be a good thing. There is just so much stuff to see and do, so much fun to be had, and that is without even bringing up the impressive multiplayer mode. Like most of the old Rare games, Tooie is also stunning on a technical level, really pushing the N64 to its limits.
Castlevania 64 & Castlevania: Legacy of Darkness
Two incredibly underrated games. I just loved the N64 Castlevania adventures, due to the killer atmosphere, lots and lots of awesome bosses, good storylines, a fantastic soundtrack and some really cool weapons. Multiple playable characters, multiple endings and various secret goodies also gave the games really good replay-value. Legacy of Darkness is definitely the better and more complete game of the two (Castlevania 64 1.5 describes it pretty well), but the original Castlevania 64 is to this day still good fun.
Rayman 2
Another third party favorite, Rayman 2 is linear 2D platforming at its very best (at least it was until Mario Galaxy came out). It is an enormously entertaining, superbly structured and well paced game full of memorable set pieces. The fact that it is one of the prettiest, most colorful N64 games available doesn't hurt either.
Star Fox 64
Quite simply a fantastic space shooter, action packed and full of excitement, memorable characters, cool bosses and awesomeness in general. The different routes to Venom and the medal-system added plenty of replay-value, while the great music gave the game a really cool atmosphere. The basic, yet very enjoyable multiplayer mode also deserves a mention.
Donkey Kong 64
Perhaps the ultimate collect-a-thon, but a dang good one. Tons of stuff to do, five playable characters with plenty of cool abilities, dozens and dozens of fun mini-games, some really cool bosses and of course, excellent graphics and sound. Even the multiplayer was pretty good.
Star Wars: Rogue Squadron
I actually like this game a lot better than the visually stunning, but overly hard and frustrating Rogue Leader for the GameCube. Anyway, this was a console Star Wars game done right, with plenty of interesting missions, all the classic ships (even a Tie Interceptor and the Naboo Starfighter) playable, a surprisingly good story, some really cool cheat codes and extremely strong replay-value due to its excellent medal-system. The atmosphere is Star Wars through and through, just the way I like it.
F-Zero X
Still one of my favorite racing games. It boasts an incredibly sense of speed, a huge number of ships and tracks, not to mention really cool course design and a 4-player mode (the SNES game didn't even have a multiplayer-mode).
Sin and Punishment
A sick mix of crazy awesome and just plain crazy (if you want an example, just look at the final boss battle), this is a 3D on-rail shooter every bit as challenging as the classic 2D ones, where kill counts in the thousands are routine and bosses are coming thick and fast. An adrenaline-pumping roller coaster ride you do not want to miss!
Those are my favorites, and no, I did not forget about the likes of Mario 64 and Goldeneye, I just simply happen to own a number of N64 games I like better than them, simple as that.