Episode 111 — Audit Data Gathering: Sampling, Questionnaires, Interviews, Assertions, and Reference Sources (5.5)

In this episode, we start looking at how audits and assessments gather evidence, because security does not become trustworthy just because someone says a control exists. An audit needs information it can review, compare, test, and explain. That information may come from samples of records, questionnaires sent to control owners, interviews with people who perform the work, assertions made by management, and reference sources that help organize what the assessment is looking for. When you are brand new to cybersecurity, the word audit may sound intimidating, but the basic idea is simple. An audit asks whether reality matches the requirement. If a policy says access must be reviewed, the audit looks for evidence that reviews actually happened. If a procedure says incidents are escalated, the audit looks for proof that escalation occurs. Evidence is what moves the conversation from belief to verification.

Before we continue, a quick note. This audio course is part of our companion study series. The first book is a detailed study guide that explains the exam and helps you prepare for it with confidence. The second is a Kindle-only eBook with one thousand flashcards you can use on your mobile device or Kindle for quick review. You can find both at Cyber Author dot me in the Bare Metal Study Guides series.

Audit data gathering begins with scope. Before anyone collects evidence, the assessment needs to know what it is reviewing and why. The scope might be a system, a department, a process, a vendor relationship, a regulation, a security control set, or a time period. Without scope, evidence collection can become scattered and unfair. You could ask for hundreds of records and still not answer the real question. A clear scope helps decide which people to interview, which documents to request, which systems to examine, and which samples to select. It also helps keep the audit from growing endlessly. For example, an assessment of account termination may focus on employees who left during a specific period and whether their access was removed on time. That is much more useful than vaguely asking whether access management seems good.

Sampling is used when reviewing every single item would be too expensive, too slow, or unnecessary. Instead of examining every account, every ticket, every change record, or every training record, the auditor selects a portion and reviews that sample. The sample should be chosen in a way that supports the purpose of the audit. Sometimes samples are random, which helps reduce bias. Sometimes they are judgmental, which means the auditor chooses items that seem especially relevant, such as high risk users, privileged accounts, critical systems, or recent incidents. Sampling does not prove that every item is perfect. It gives a reasonable view of whether the process is working. If a sample shows repeated failures, the auditor may expand the review or conclude that the control is not operating effectively. Sampling helps make large environments reviewable.

Good sampling depends on understanding the population being sampled. The population is the complete group of items that could be selected. If the audit is reviewing terminated employee access, the population might be all employees who left during a certain time period. If the audit is reviewing firewall change approvals, the population might be all firewall changes during a quarter. If the population is wrong, the sample will be weak. Imagine trying to review access removals but receiving only a list of people from one department when the organization has many departments. The sample may look clean while the real risk is hiding elsewhere. This is why data quality matters during audit work. The auditor needs confidence that the list of items is complete enough to support the review. Strong evidence begins with knowing what universe of records is being examined.

Questionnaires are another common way to gather audit data. A questionnaire asks structured questions so the auditor can collect consistent information from different people, teams, systems, or vendors. It may ask whether a control exists, who owns it, how often it is performed, what evidence is retained, what tools support it, and whether exceptions are allowed. Questionnaires are useful because they save time and create a record of responses. They are especially common when reviewing many teams or third parties. The risk is that people may answer based on what they believe should happen instead of what actually happens. A team may say access is reviewed quarterly, but the evidence may show the last review was incomplete or never approved. A questionnaire is a starting point. It should be supported by documents, records, screenshots, logs, tickets, or interviews when the risk is important.

A strong questionnaire uses plain questions and avoids traps. The goal is not to confuse the person answering. The goal is to understand the process accurately. Questions should be specific enough to produce useful answers. Asking whether security is managed well is too broad. Asking who approves privileged access, how approval is recorded, and how often privileged access is reviewed is much clearer. Good questionnaires also leave room for explanation. A yes or no answer may not capture reality when a process is partly manual, changing, or different across systems. For a new security professional, it helps to read questionnaire responses with a careful mind. Do not assume that a confident answer is enough. Ask what evidence would show the answer is true. The evidence is what turns a response into something the audit can rely on.

Interviews help auditors understand how processes work in real life. Documents may say one thing, but people who perform the work can explain what actually happens when requests arrive, systems fail, alerts trigger, or exceptions appear. Interviews are especially useful for understanding responsibility, judgment, escalation, and informal workarounds. A policy may say all changes require approval, but an interview may reveal that emergency changes are sometimes approved after the fact. That may be acceptable if documented and controlled, or it may reveal a weakness. Interviews also help identify missing evidence. A control owner might explain that evidence exists in a ticketing system, while another person might explain that a report is produced but not retained. The interview does not replace written proof, but it gives context that helps the auditor know what to request and how to interpret what is found.

Good interviews are respectful, focused, and evidence oriented. The auditor is not there to embarrass people or make them feel trapped. A good interview should help uncover the truth of the process. People are more likely to explain real problems when the conversation is professional and clear. The auditor may ask what the person does, how often they do it, what tools they use, what records are created, how exceptions are handled, and what problems they see. This can reveal control gaps that documents never mention. For example, a team might have a strong written incident process, but an interview may reveal that only one person knows how to contact a critical vendor after hours. That kind of information matters because security depends on real human workflow. Interviews help connect written requirements to lived operation.

Assertions are statements made by management, control owners, vendors, or other responsible parties about the condition of a process, system, or control. An assertion might say that backups are tested monthly, access reviews are completed quarterly, logs are retained for a required period, or sensitive data is encrypted. Assertions matter because someone with responsibility is formally claiming that something is true. However, an assertion alone is not the same as independent verification. An auditor may use assertions to understand what management believes is happening, but then compare those assertions to evidence. If management asserts that all terminated users are removed within twenty four hours, the auditor may sample termination records and account logs to see whether that happened. Assertions help frame the review, but evidence determines whether the assertion is supported.

There are different kinds of assertions that can show up in audit thinking. An assertion may relate to completeness, meaning all required items are included. It may relate to accuracy, meaning the information is correct. It may relate to existence, meaning the control or record actually exists. It may relate to occurrence, meaning the activity actually happened. It may relate to authorization, meaning the right person approved the action. You do not need to memorize a long accounting-style list for this episode, but you should understand the idea. Assertions are claims that need support. If a report claims every privileged account was reviewed, the auditor may ask whether the report includes all privileged accounts, whether the review happened during the required period, whether exceptions were handled, and whether approval was recorded. That turns a general claim into testable evidence.

Reference sources help auditors and assessors decide what to look for and how to organize findings. One well-known source is MITRE Adversarial Tactics, Techniques, and Common Knowledge (A T T and C K). It organizes common attacker behaviors into tactics and techniques, which helps security teams think about how real intrusions unfold. Another reference source is the Cyber Kill Chain, which describes stages an attacker may move through, from early preparation through actions on objectives. The Diamond Model is another way to analyze cyber events by looking at the relationship among adversary, capability, infrastructure, and victim. These sources are not audit evidence by themselves. They are frameworks that help assessors ask better questions, map observations, and understand whether defenses and detections cover realistic attacker behavior. They help turn scattered security details into a more organized view of risk.

Using reference sources wisely means understanding their purpose and their limits. A T T and C K can help an assessment ask whether monitoring covers credential theft, persistence, lateral movement, or command and control activity. The Cyber Kill Chain can help show where controls may detect or disrupt an attack path. The Diamond Model can help analysts connect what happened, who may have acted, what capability was used, what infrastructure supported the activity, and who was targeted. These models can strengthen audit and assessment work because they keep the review from focusing only on checkboxes. Still, they should not be forced into every situation. A privacy retention audit may not need an attacker behavior framework. A detection engineering review might benefit from one. Good assessors choose reference sources that fit the question being asked.

Audit evidence should be relevant, reliable, and sufficient. Relevant evidence answers the question being asked. Reliable evidence comes from a trustworthy source and has not been casually altered. Sufficient evidence means there is enough information to support a conclusion. A single interview may be relevant, but not sufficient for a high risk control. A system-generated log may be more reliable than a memory-based answer, but it still needs to cover the right time period and the right system. Evidence can include policies, procedures, logs, tickets, configuration records, screenshots, approvals, reports, training records, contracts, attestations, and test results. As you listen to audit questions on the exam, keep asking what would prove the claim. If the claim is that a process happened, look for records. If the claim is that a requirement exists, look for the approved document.

The main idea to carry forward is that audit data gathering is about moving from statements to supported conclusions. Sampling helps assess large sets of records without reviewing every item. Questionnaires collect structured information, but their answers need support. Interviews reveal how processes work in real life and help explain gaps, exceptions, and context. Assertions are claims made by responsible parties, and audits compare those claims against evidence. Reference sources such as A T T and C K, the Cyber Kill Chain, and the Diamond Model help organize security thinking and guide better questions when they fit the assessment. For Security Plus S Y Zero Eight Zero One, remember that audits are not only about catching mistakes. They help the organization understand whether controls are designed, operating, documented, and supported by evidence that can be trusted.

Episode 111 — Audit Data Gathering: Sampling, Questionnaires, Interviews, Assertions, and Reference Sources (5.5)
Broadcast by