I have an issue where when I start up modded minecraft, PC uses little ram and system monitor reports that java never exceeds the max of 8gb ram I set. When I start flying around and mapping out the server using a minimap and world map, RAM usage keeps rising till PC starts lagging showing 90%+ of ram being used despite system monitor reporting 8gb for java and mb (or less) of usage on everything else. I even tried using free and htop in terminal but same results. Closing minecraft makes ram drop, but never all the way as shown in the images.

I am on Linux Mint 21.2, running minecraft through prism launcher, modpack was ATM8, but I noticed it happens on all modpacks. Not sure if this is a minecraft or mint issue, so I hope someone can explain or point me in the right direction