Troubleshooting

No data is added to the database

Check the $ENTRADA_LOG_DIR/ENTRADA-partial-loader.log logfile for any errors. There might be a problem with the ENTRADA configuration.

Pause processing

Temporarily stopping the pcap processing can be done by disabling the cron job for the pcap conversion process (run_02_partial_loader_bootstrap.sh).

Old pcap files take up too much space

Decrease the value of the PCAP_DAYS_TO_KEEP configuration option, the default value is 10 days which might be too long if you process a lot of traffic and have a small local data disk.

Re-process data

If old pcap files need to re-processed because the Parquet files on HDFS have been lost for some reason. Then move the pcap files from the $DATA_DIR/processed/archive/<server>/ directory to the $DATA_DIR/incoming/<server>/ directory. The files will automatically be re-processed.

Modify database rows

This is not possible, the data is written once as immutable Parquet formatted files.

Remove data from a table

The data is stored as Parquet files on HDFS and the Impala tables point to these files. To delete data from the database the data files must be removed from hdfs://$HDFS_HOME/<table_directory>/ and the corresponding Impala partitions need to be removed with the following SQL code.

ALTER TABLE ... DROP PARTITION ...