My environment is:

• RelStorage = 4.0.0
• PostgreSQL = 15.6
• ZODB = 5.8.1

What happened: i tried to pack a large db with blobs after some months of working.

I tried to do it several times without success, because there were some problems that stopped the procedure:

• not enough permissions on tables or on large objects (the dbadmins solved setting lo_compat_privileges = on)
• server run out of space because of too many wal files
• server run out of resources because of too big transactions

To solve some resource problems started launching packs with different days (40, 30, 20, 10, 0).

Every time that i had a problem and i solved it, i restarted the pack without cleaning any table on the db.

The last time that i restarted it, i've found that the db were corrupted and all the blobs returned a POSKeyError.
Also after the pack has succeeded, the db had all the blobs broken.

A backup made before last pack (the one that breaks blobs), the db is ok. the db also went from 140gb to 40.

I've also tried to launch the pack on that backup but i can't reproduce the error (only once stopping the pack several times, but i can't reproduce it).

Any ideas on what could get wrong?

This is an example of traceback with errors:

2024-07-03 09:05:59,807 ERROR [ZODB.Connection:804][MainThread] Couldn't load state for Persistence.mapping.PersistentMapping 0x0e6b85
Traceback (most recent call last):
File "/opt/plone/eggs/ZODB-5.8.1-py3.11.egg/ZODB/Connection.py", line 787, in setstate
p, serial = self._storage.load(oid)
^^^^^^^^^^^^^^^^^^^^^^^
File "src/perfmetrics/metric.py", line 72, in perfmetrics._metric._AbstractMetricImpl.call
File "/opt/plone/eggs/RelStorage-4.0.0-py3.11-linux-x86_64.egg/relstorage/storage/load.py", line 109, in load
raise self.__pke(oid,
relstorage.interfaces.POSKeyError: 0x0e6b85 ({'cache': <StorageCache at 0x7fcc16a119a0 hvt=286454828400357461 bytes=9842757 len=16271 state=<relstorage.cache.mvcc.MVCCDatabaseCoordinator object at 0x7fcc1693b690>>, 'load': <LoadConnection at 0x7fcc12e81cd0 active=True description={'backend_pid': 2382722} conn=<connection object at 0x7fcc13363ec0; dsn: 'user=xxx password=xxx dbname=xxx host=xxx port=5432 application_name='RS: Load'', closed: 0> cur=<cursor object at 0x7fcc12fe64d0; closed: 0>>, 'adapter': <PostgreSQLAdapter at 7fcc12e81d50 history preserving,dsn="dbname='xxx' host='xxx' port='5432' user='xxx'">, 'reason': 'no tid found', 'current_txn': 286454828400357461, 'recent_tids': '945029'})

2024-07-03 09:05:59,823 ERROR [Zope.SiteErrorLog:36][MainThread] POSKeyError: https://xxx/++api++/@secondary-menu
Traceback (innermost last):
Module ZPublisher.WSGIPublisher, line 181, in transaction_pubevents
Module ZPublisher.WSGIPublisher, line 391, in publish_module
Module ZPublisher.WSGIPublisher, line 285, in publish
Module ZPublisher.mapply, line 98, in mapply
Module ZPublisher.WSGIPublisher, line 68, in call_object
Module plone.rest.service, line 21, in call
Module plone.restapi.services, line 19, in render
Module collective.volto.secondarymenu.restapi.secondary_menu.get, line 28, in reply
Module collective.volto.secondarymenu.restapi.serializer.secondary_menu, line 36, in serialize_data
Module design.plone.ctgeneric.restapi.serializer.summary, line 20, in call
Module design.plone.contenttypes.restapi.serializers.summary, line 152, in call
Module redturtle.volto.restapi.serializer.summary, line 72, in call
Module plone.restapi.serializer.summary, line 99, in call
Module plone.app.contentlisting.realobject, line 89, in review_state
Module Products.CMFCore.WorkflowTool, line 283, in getInfoFor
Module Products.DCWorkflow.DCWorkflow, line 300, in getInfoFor
Module Products.DCWorkflow.DCWorkflow, line 130, in _getWorkflowStateOf
Module Products.CMFCore.WorkflowTool, line 341, in getStatusOf
Module Products.CMFCore.WorkflowTool, line 625, in get
Module ZODB.Connection, line 787, in setstate
Module perfmetrics._metric, line 72, in perfmetrics._metric._AbstractMetricImpl.call
Module relstorage.storage.load, line 109, in load
relstorage.interfaces.POSKeyError: 0x0e6b85 ({'cache': <StorageCache at 0x7fcc16a119a0 hvt=286454828400357461 bytes=9843160 len=16272 state=<relstorage.cache.mvcc.MVCCDatabaseCoordinator object at 0x7fcc1693b690>>, 'load': <LoadConnection at 0x7fcc12e81cd0 active=True description={'backend_pid': 2382722} conn=<connection object at 0x7fcc13363ec0; dsn: 'user=xxx password=xxx dbname=xxx host=xxx port=5432 application_name='RS: Load'', closed: 0> cur=<cursor object at 0x7fcc12fe64d0; closed: 0>>, 'adapter': <PostgreSQLAdapter at 7fcc12e81d50 history preserving,dsn="dbname='xxx' host='xxx' port='5432' user='xxx'">, 'reason': 'no tid found', 'current_txn': 286454828400357461, 'recent_tids': '945029'})

2024-07-03 09:06:18,673 ERROR [ZODB.Connection:804][MainThread] Couldn't load state for BTrees.IIBTree.IITreeSet 0x1789e1
Traceback (most recent call last):
File "/opt/plone/eggs/ZODB-5.8.1-py3.11.egg/ZODB/Connection.py", line 787, in setstate
p, serial = self._storage.load(oid)
^^^^^^^^^^^^^^^^^^^^^^^
File "src/perfmetrics/metric.py", line 66, in perfmetrics._metric._AbstractMetricImpl.call
File "/opt/plone/eggs/RelStorage-4.0.0-py3.11-linux-x86_64.egg/relstorage/storage/load.py", line 109, in load
raise self.__pke(oid,
relstorage.interfaces.POSKeyError: 0x1789e1 ({'cache': <StorageCache at 0x7fcc16a119a0 hvt=286454828400357461 bytes=9843077 len=16330 state=<relstorage.cache.mvcc.MVCCDatabaseCoordinator object at 0x7fcc1693b690>>, 'load': <LoadConnection at 0x7fcc12e81cd0 active=True description={'backend_pid': 2382722} conn=<connection object at 0x7fcc13363ec0; dsn: 'user=xxx password=xxx dbname=xxx host=xxx port=5432 application_name='RS: Load'', closed: 0> cur=<cursor object at 0x7fcc12fe64d0; closed: 0>>, 'adapter': <PostgreSQLAdapter at 7fcc12e81d50 history preserving,dsn="dbname='xxx' host='xxx' port='5432' user='xxx'">, 'reason': 'no tid found', 'current_txn': 286454828400357461, 'recent_tids': '1542625'})