Partitioning

The database tables use a partitioning schema to divide the data into separate partition. These partitions MUST be used in SQL queries to indicate which data files should be used when executing the SQL query, this process is called partition pruning. The partition schema contains 4 columns.

Column Description
year The capture year
month The capture month
day The capture day
server The destination server

The following example SQL query only analyses data for server “ns1.dns.nl” that was captured on 05-12-2015. All other data in the table is skipped. Partitioning functions as an index to enable fast data lookups.

Using partition pruning the query engine can ignore most of the database data and this will save a lot of processing time and money in the case of Athena ( for Athena you pay per bytes scanned)

select qname
from dns.queries
where year=2015 and month=12 and day=5 and server="ns1.dns.nl"
limit 10