Fetch data from customer owned S3 bucket into Eon DB

If you grant access to the customer bucket in the IAM role used for the EC2 instances the Vertica database will be able to read the bucket directly.

OK, I had tried that using directions customer recommended at https://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html ("Follow the Account B path", they said)
Now I can list their bucket, but I seem to have broken permission on my own bucket such that Vertica can no longer commit depot or catalog files to communal storage! Neither SAL ALL logging nor CloudTrail really give an idea what I broke. Any thoughts on how to debug S3 policy or permissions? (FWIW, the catalog file listed below exists, but from several days ago)
Here is vertica.log:
2018-06-08 03:39:40.106 Init Session:7f306229b700 [Session] [Query] TX:0(v_verticadb_node0001-20856:0x4e4d7) select sync_catalog();
2018-06-08 03:39:40.109 Init Session:7f306229b700-b00000000176d3 [Txn] Begin Txn: b00000000176d3 'static void TXN::TransAPI::updateClusterInfoWithTruncationVersion(bool, bool)'
2018-06-08 03:39:40.109 Init Session:7f306229b700-b00000000176d3 [Txn] Polling all nodes in cluster for latest catalog version (sync catalog on all nodes? Yes) (is this shutdown? No)
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [Txn] Considering Checkpoint
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [SAL] Adding (txn=b00000000176d3,statement=1) to HDFS cancel callback monitoring
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [Txn] [TxnLogSyncTask] Determining Txn logs to sync to s3://verticavimeodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0001/Catalog/f5e5c377afd44a5f89f40c3c5
bfe9e
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [SAL] [S3UDFS] File opened: s3://verticavimeodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0001/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7494_g748
1.cat
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [SAL] [S3UDFS] Found a global client for bucket 'verticavimeodata'
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [SAL] [S3UDFS] S3FileCancelHandle :: Adding (txn=b00000000176d3,statement=1) to S3FS cancel callback monitoring
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [SAL] [S3UDFS] [s3://verticavimeodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0001/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7494_g7481.cat]: virt
ual Vertica::UDFileOperator* SAL::S3FileSystem::open(const char*, int, mode_t, bool) const, Opened file
2018-06-08 03:39:40.111 Init Session:7f306229b700-b00000000176d3 [SAL] S3FileOperator::append() [s3://verticavimeodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0001/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7494
_g7481.cat] bytes = 1048576, upload buffer size = 26214400
2018-06-08 03:39:40.161 Init Session:7f306229b700-b00000000176d3 [SAL] [S3UDFS] [s3://verticavimeodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0001/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7494_g7481.cat]: virt
ual void SAL::S3FileSystem::close(Vertica::UDFileOperator*) const, Closed file
2018-06-08 03:39:40.161 Init Session:7f306229b700-b00000000176d3 [SAL] [S3UDFS] [s3://verticavimeodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0001/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7494_g7481.cat]: void
SAL::S3FileOperator::finalize(),
2018-06-08 03:39:40.165 Init Session:7f306229b700-b00000000176d3 [SAL] Removing (txn=b00000000176d3,statement=1( from HDFS cancel callback monitoring
2018-06-08 03:39:40.178 Init Session:7f306229b700-b00000000176d3 @v_verticadb_node0003: 42501/8076: Could not copy file [/vertica/data/VerticaDB/v_verticadb_node0003_catalog/Catalog/Txnlogs/txn_7485_g7481.cat] to [s3://verticavim
eodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0003/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7485_g7481.cat]: Access Denied
LOCATION: copyTo, /scratch_a/release/svrtar30992/vbuild/vertica/Util/IO/File.cpp:480
2018-06-08 03:39:40.178 Init Session:7f306229b700-b00000000176d3 @v_verticadb_node0001: 42501/8076: Could not copy file [/vertica/data/VerticaDB/v_verticadb_node0001_catalog/Catalog/Txnlogs/txn_7494_g7481.cat] to [s3://verticavim
eodata/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0001/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7494_g7481.cat]: Access Denied
LOCATION: copyTo, /scratch_a/release/svrtar30992/vbuild/vertica/Util/IO/File.cpp:480
2018-06-08 03:39:40.178 Init Session:7f306229b700-b00000000176d3 [Txn] Rollback Txn: b00000000176d3 'static void TXN::TransAPI::updateClusterInfoWithTruncationVersion(bool, bool)'
2018-06-08 03:39:40.179 Init Session:7f306229b700 @v_verticadb_node0002: 42501/8076: Could not copy file [/vertica/data/VerticaDB/v_verticadb_node0002_catalog/Catalog/Txnlogs/txn_7494_g7481.cat] to [s3://verticavimeodata/eon/data
/metadata/VerticaDB/nodes/v_verticadb_node0002/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7494_g7481.cat]: Access Denied
LOCATION: copyTo, /scratch_a/release/svrtar30992/vbuild/vertica/Util/IO/File.cpp:480
2018-06-08 03:39:41.200 Init Session:7f306229b700 @v_verticadb_node0001: 00000/4719: Session v_verticadb_node0001-20856:0x4e4d7 ended; closing connection (connCnt 8)

You can't set AWSAuth to a key that doesn't have permissions on the eon database communal storage. Otherwise you get the behavior you are observing. I would suggest clearing the parameter (ALTER DATABASE DEFAULT CLEAR AwsAuth).
You can try setting that parameter at the session level instead (ALTER SESSION SET AwsAuth = ...).

I figured as much since the S3 bucket logs showed attempted auth as the customer's IAM role. However, I did what you suggested and got an error I didn't expect: it seems that node0003 still can't sync_catalog(), though I checked and node0001 and node0002 worked. Also, I was able to copy this file using "aws s3 cp" from command line on node0003. Any thoughts on the following:
dbadmin=> ALTER DATABASE DEFAULT CLEAR AwsAuth;
ALTER DATABASE
dbadmin=> ALTER SESSION SET AWSAuth='key:secret';
ALTER SESSION
dbadmin=> select /+LABEL(zyzzx3)/ sync_catalog();
ERROR 8076: Could not copy file [/vertica/data/VerticaDB/v_verticadb_node0003_catalog/Catalog/Txnlogs/txn_7485_g7481.cat] to [s3://a_random_bucket/eon/data/metadata/VerticaDB/nodes/v_verticadb_node0003/Catalog/f5e5c377afd44a5f89f40c3c5bfe9e/Txnlogs/txn_7485_g7481.cat]: Access Denied

Please delete your key from the above post!

I wouldn't be trying to run sync_catalog() in a session with AWSAuth set. The question is whether you can run COPY t from 's3://newbucket/whatever.csv'

Yes, with AWSAuth set, I can read the Eon communal storage, where I've placed a source folder separate from the communal storage. Re: catalog, should I just watch the node0003 log for the regular 5 minute update and see what happens? And re: IAM role, does that mean the customer needs to grant the IAM role access to e.g. ListBucket and GetObject on their S3 bucket? I sent them the ARN of my role already but would like to verify that is heading in the correct direction. Thanks!

We have it mostly working now with explicit inline policies granting bucket access both to the user associated with the AWSAuth keys as well as the EC2 instance role. This may be overkill but we're not touching it anymore! Probable significant factors are granting the exact same functions on the bucket, role, user, as well as the exact same pattern of bucket and objects in each JSON entry.