cross-posted from: https://beehaw.org/post/282116

We’ve posted a number of times about our increasing storage issues. We’re currently at the cusp of using 80% of the 25gb we have available in the current tier for the online service we run this instance on. This has caused some issues with the server crashing in recent days.

We’ve been monitoring and reporting on this progress occasionally, including support requests and comments on the main lemmy instance. Of particular note, it seems that pictures tend to be the culprit when it comes to storage issues.

The last time a discussion around pict-rs came up, the following comment stuck out to me as a potential solution

Storage requirements depend entirely on the amount of images that users upload. In case of slrpnk.net, there are currently 1.6 GB of pictrs data. You can also use s3 storage, or something like sshfs to mount remote storage.

Is there anyone around who is technically proficient enough to help guide us through potential solutions using “something like sshfs” to mount remote storage? As it currently exists, our only feasible option seems to be upgrading from $6/month to $12/month to double our current storage capacity (25GB -> 50 GB) which seems like an undesirable solution.

  • Gaywallet (they/it)
    3•1 year ago

    I am willing to contribute storage (I have several TB), but I am somewhat bandwidth limited, so I need to be a bit careful with hosting too many images to not impact the other services that I run on the same connection.

    How would you accomplish this? I have plenty of bandwidth and plenty of storage I can subsection as a possible solution (hell even buying a raspberry pi and an old hard drive wouldn’t be all that expensive and potentially a fun project) but I really don’t even have an idea of how to connect this to the lemmy instance

    • poVoqA
      2•1 year ago

      See the link above. You can configure Pict-rs 4.0 (beta, unreleased) to a S3 compatible storage. The Garage project is a S3 compatible storage especially aimed at distributed self-hosters, but with some latency caveats aside something like Minio would probably also work.

      S3 storage allows to redirect users directly to a storage location (with a public IP) instead of the main server loading images from a storage location and serving that to the user itself. Kind of like a CDN works.

      • Gaywallet (they/it)
        1•1 year ago

        Is there any way to do this and avoid having to use S3? I don’t want a surprise bill from Amazon because we exceeded some thresholds they have on the free tier (nor do I want to have to make new free tiers every 12 months).

          • Gaywallet (they/it)
            1•1 year ago

            Okay so I need to be sure I have something that can make sense of s3 calls to storage, I feel like we’re getting closer, just still way out of my own technological depth.

            • poVoqA
              3•1 year ago

              Pict-rs that Lemmy uses as the image storage is able to do S3 compatible storage API calls in the upcoming 4.0 version.

              There are also many self-hosted options that you can install on your server to provide a S3 compatible storage API. The probably best known open-source software for that is called Minio. It is however more meant for data-centers with fast low latency connections or local network only.

              The above mentioned Garage software is unique in that it is specifically designed to work in less than ideal networking conditions typical to self-hosted servers.