+

improvement/s3-bucket-storage#1

Merged

117 commits merged intomasterfrom

improvement/s3-bucket-storage

Oct 1, 2021

Commits on Sep 15, 2021

Enable s3 bucket storage. Listkeys uses s3fs while S3_FSCK uses native spark s3a filsystem access.

trevorbenson
authored and
trevorbenson
committed
Changes for Dockefiles slimming container, including jars required for s3a filesystem access, updates to config template to handle srebiuldd changes to arc driver paths

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 17, 2021

First pass at obtaining the arc data keys metadata to extract the object key.

trevorbenson
authored and
trevorbenson
committed
implement struct.unpack to access the object key.

trevorbenson
authored and
trevorbenson
committed
Added missing import

trevorbenson
authored and
trevorbenson
committed
Adjust data.append to inside loop

trevorbenson
authored and
trevorbenson
committed
Fix indentation

trevorbenson
authored and
trevorbenson
committed
Missing base64 module

trevorbenson
authored and
trevorbenson
committed
Indentation again from pasting from scratchpad.

trevorbenson
authored and
trevorbenson
committed
print to diagnose where match is not working.

trevorbenson
authored and
trevorbenson
committed
print to diagnose where match is not working.

trevorbenson
authored and
trevorbenson
committed
unstringify the regex search bariable

trevorbenson
authored and
trevorbenson
committed
extra print to see if pattern ever matches.

trevorbenson
authored and
trevorbenson
committed
change regex search to if not to isolate why regex is not matching.

trevorbenson
authored and
trevorbenson
committed
change regex search to if not to isolate why regex is not matching.

trevorbenson
authored and
trevorbenson
committed
change regex search to if not to isolate why regex is not matching.

trevorbenson
authored and
trevorbenson
committed
fix CHUNK_STATUS_OK to CHUNKAPI_STATUS_OK.

trevorbenson
authored and
trevorbenson
committed
Fixing mixed indentation lengths.

trevorbenson
authored and
trevorbenson
committed
debug cleanup

trevorbenson
authored and
trevorbenson
committed
Fixing mixed indentation lengths

trevorbenson
authored and
trevorbenson
committed
Fixing mixed indentation lengths

trevorbenson
authored and
trevorbenson
committed
Fixing mixed indentation lengths

trevorbenson
authored and
trevorbenson
committed
Fixing mixed indentation lengths

trevorbenson
authored and
trevorbenson
committed
Cleanup prints for debugging.

trevorbenson
authored and
trevorbenson
committed
Update to documentation

trevorbenson
authored and
trevorbenson
committed
Update to documentation

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 18, 2021

normalizing cfg/cmdline args to uppercase. Using same double quote style. Documentation mentions arc_protection requirement.

trevorbenson
authored and
trevorbenson
committed
Fixing indentation after normalization

trevorbenson
authored and
trevorbenson
committed
Fixing indentation after normalization

trevorbenson
authored and
trevorbenson
committed
Implementing protection and arcdatakeypattern to calculate the accurate space consumed by non indexed keys.

trevorbenson
authored and
trevorbenson
committed
Implementing protection and arcdatakeypattern to calculate the accurate space consumed by non indexed keys.

trevorbenson
authored and
trevorbenson
committed
Implementing protection and arcdatakeypattern to calculate the accurate space consumed by non indexed keys.

trevorbenson
authored and
trevorbenson
committed
unmasking builtin all()

trevorbenson
authored and
trevorbenson
committed
unmasking builtin all()

trevorbenson
authored and
trevorbenson
committed
reducing redundant path name components

trevorbenson
authored and
trevorbenson
committed
reducing redundant path name components

trevorbenson
authored and
trevorbenson
committed
Normalizing

trevorbenson
authored and
trevorbenson
committed
Normalizing

trevorbenson
authored and
trevorbenson
committed
Normalizing

trevorbenson
authored and
trevorbenson
committed
Normalizing

trevorbenson
authored and
trevorbenson
committed
Normalizing

trevorbenson
authored and
trevorbenson
committed
Normalizing

trevorbenson
authored and
trevorbenson
committed
Reverting the regex search beck

trevorbenson
authored and
trevorbenson
committed
dataframe.show to debug carrying ip and chordport forward.

trevorbenson
authored and
trevorbenson
committed
dataframe.show to debug carrying ip and chordport forward.

trevorbenson
authored and
trevorbenson
committed
dataframe.show to debug carrying ip and chordport forward.

trevorbenson
authored and
trevorbenson
committed
dataframe.show to debug carrying ip and chordport forward.

trevorbenson
authored and
trevorbenson
committed
dataframe.show to debug carrying ip and chordport forward.

trevorbenson
authored and
trevorbenson
committed
Implement join to bring object key into dftotal

trevorbenson
authored and
trevorbenson
committed
Implement join to bring object key into dftotal

trevorbenson
authored and
trevorbenson
committed
Implement join to bring object key into dftotal

trevorbenson
authored and
trevorbenson
committed
Implement join to bring object key into dftotal

trevorbenson
authored and
trevorbenson
committed
Implement join to bring object key into dftotal

trevorbenson
authored and
trevorbenson
committed
Implement join to bring object key into dftotal

trevorbenson
authored and
trevorbenson
committed
Swap to key = row._c1 to delete object key, not arc data key.

trevorbenson
authored and
trevorbenson
committed
Swap to key = row._c1 to delete object key, not arc data key.

trevorbenson
authored and
trevorbenson
committed
Swap to key = row._c1 to delete object key, not arc data key.

trevorbenson
authored and
trevorbenson
committed
Swap to key = row._c1 to delete object key, not arc data key.

trevorbenson
authored and
trevorbenson
committed
Adding s3fs to worker

trevorbenson
authored and
trevorbenson
committed
adding whl so container building is supported after repo clone.

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 19, 2021

path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 20, 2021

path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix and objectkey join in p4

trevorbenson
authored and
trevorbenson
committed
path prefix documentation

trevorbenson
authored and
trevorbenson
committed
legacy path cleanup

trevorbenson
authored and
trevorbenson
committed
slimming, pip install s3fs for cluster run listkeys, layer reductions

trevorbenson
authored and
trevorbenson
committed
workflow diagram

trevorbenson
authored and
trevorbenson
committed
workflow diagram

trevorbenson
authored and
trevorbenson
committed
Partitions calculated by executor: instances * cores. Debug prints in p3/p4

trevorbenson
authored and
trevorbenson
committed
Partitions calculated by executor: instances * cores. Debug prints in p3/p4

trevorbenson
authored and
trevorbenson
committed
remove extra print status and store NOK for failure in REVLOOKUPID

trevorbenson
authored and
trevorbenson
committed
remove extra print status and store NOK for failure in REVLOOKUPID

trevorbenson
authored and
trevorbenson
committed
Update readme for s3_fsck_pX files.

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 21, 2021

Update readme for s3_fsck_pX files and warning on changing cluster execution commands.

trevorbenson
authored and
trevorbenson
committed
Update readme for s3_fsck_pX files and warning on changing cluster execution commands.

trevorbenson
authored and
trevorbenson
committed
Readme script directory/name fixes.

trevorbenson
authored and
trevorbenson
committed
dataframe.index not working as described

trevorbenson
authored and
trevorbenson
committed
Debug prints to isolate where join drops 90% of keys.

trevorbenson
authored and
trevorbenson
committed
df.show() has no len(), reverting debug prints.

trevorbenson
authored and
trevorbenson
committed
remove filter and drop to diagnose why keep ending up with 50 keys deleted.

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 23, 2021

Adding arc_protection to the config-template.yml, used by S3_FSCK. After PR merged root README.md should have example as well as SOFS_FSCK scripts need to be updated to set from config.

trevorbenson
authored and
trevorbenson
committed
Documentation update which shows how to stream the keys.txt file into the bucket without storing it first.

trevorbenson
authored and
trevorbenson
committed
Documentation update which shows how to stream the keys.txt file into the bucket without storing it first.

trevorbenson
authored and
trevorbenson
committed
Documentation update which shows how to stream the keys.txt file into the bucket without storing it first.

trevorbenson
authored and
trevorbenson
committed
Documentation update which shows how to stream the keys.txt file into the bucket without storing it first.

trevorbenson
authored and
trevorbenson
committed
Documentation update which shows how to stream the keys.txt file into the bucket without storing it first.

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 29, 2021

BREAKING CHANGES: config.yml parameters renamed!

trevorbenson
authored and
trevorbenson
committed
BREAKING CHANGES: config.yml parameters renamed!

trevorbenson
authored and
trevorbenson
committed
BREAKING CHANGES: config.yml parameters renamed!

trevorbenson
authored and
trevorbenson
committed

Commits on Sep 30, 2021

Write via dataframes to the bucket.

trevorbenson
authored and
trevorbenson
committed
Unify filename/path for both protocols and print node status from listkeys as no longer a dataframe.

trevorbenson
authored and
trevorbenson
committed

点击这是indexloc提供的php浏览器服务，不要输入任何密码和下载