I have a large object that I want to save to the disk because it takes a minute to generate. The OOM reaper kills the process while pickle.dump ing the object.

It’s a tuple of dicts of tuple of array.array.

Can pickle dump in chunks? If not, is there another technique I can use?

  • Biorix@lemmy.fmhy.ml
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    How are you dumping it?

    Can you show us the code?

    Have you tried splitting your tuple and save each dicts of the tuple separately?