cmc/cleberg.net

  1#+date:        [2025-11-23 Sun 17:43:57]
  2#+title:       IT Audit and AI: Where It Helps and Where It's Theater
  3#+description: How I'm currently using AI in my Technology Assurance role at KPMG.
  4#+slug: it-audit-ai
  5#+filetags:    :audit:
  6
  7* My AI Toolkit
  8
  9As others in the Big 4 will probably tell you, AI is a strategic tool that
 10allows for a far more efficient and impact audit. However, the key is
 11understanding when to use it, how to use it, and how to translate the output
 12from your AI into actions.
 13
 14This post will walk you through how I personally leverage AI in my role.
 15
 16Let's start by stepping through the tools I have at my disposal.
 17
 18** Gemini Enterprise (formerly Agentspace)
 19
 20First and foremost is [[https://cloud.google.com/gemini-enterprise][Gemini Enterprise]], which has a large context window, is
 21connected to the Enterprise's internal documentation and websites, and allows
 22for specialized use of custom agents, [[https://notebooklm.google/][NotebookLM]], and more.
 23
 24I use Gemini for a handful of specific use-cases, notably:
 25
 261. NotebookLM notebooks for team-based usage. For example, you can build a
 27   notebook for a specific client (using public and private information about
 28   that client) and then chat with the data in the notebook, generate an audio
 29   overview, and save documents, such as mind maps or study guides, for future
 30   use.
 31   - I love this because it provides a "run once and save" functionality, where
 32     I can generate things like an audio overview or document, save it, and then
 33     my team can reference it without re-running the prompt or even needing to
 34     open the notebook, if they download it.
 352. Custom Agents for customized personas that benefit from the private
 36   information that Gemini has access to, such as internal SharePoint sites.
 37   - For example, I have a custom agent designed to operate as an IT Audit
 38     Assistant that can help you ask questions that can be answered by the
 39     company's news, documentation sites, and more (e.g., "I'm testing logical
 40     access for AWS and want to know if anyone has created a solution or
 41     guidance for XYZ.").
 42
 43** Microsoft Copilot
 44
 45Likely the most common AI in use at companies today, due to its integration with
 46every Microsoft product, Copilot offers the best AI solution for productivity.
 47However, it's usage is likely also the most volatile due to the large gap in
 48skill sets between new-hires and senior employees.
 49
 50For associates and other early-career employees, I suggest that they use Copilot
 51to perform two primary tasks:
 52
 531. Check their tone, such as making an email or messages more professionalism,
 54   softer, or perhaps to analyze the conversation up to that point to understand
 55   how to respond better. This is something that /should/ come naturally to
 56   senior employees, but is often a learning curve for associates.
 572. For written documentation, such as guides or presentations, use Copilot and
 58   provide it with ample information on the goal of the documentation in order
 59   for Copilot to suggest an outline, gaps in the writer's thinking, or areas
 60   for further research before writing the document. The key here is to use
 61   Copilot to improve your thinking and not to replace it. It shouldn't generate
 62   the content for you, it should remove the tedium of creating boilerplate
 63   structures or to design the structure of a document.
 64
 65** Clara AI
 66
 67[[https://kpmg.com/us/en/capabilities-services/audit-services/kpmg-clara.html][Clara AI]] is KPMG's audit-based AI platform, known as the "KPMG Clara Smart Audit
 68Platform". The major benefit to Clara AI is that it lives within a sandboxed
 69environment, running on white-labelled models, fine-tuned on firm-specific
 70methodologies, and built directly into KPMG's audit tools.
 71
 72As noted in KPMG's Clara AI page linked above, Clara AI is built directly into
 73the Clara audit platform in order to provide systematic risk assessments, drive
 74consistent and efficient audits by providing all audit teams with relevant
 75information for their audit, and connects to other tools (such as KPMG Client
 76Collaboration, where evidence is requested and provided).
 77
 78I *absolutely love this tool*, as it solves some of the bigger problems for
 79auditors at large firms:
 80
 811. Connecting younger staff to the firm's guidance and regulatory material,
 82   lowering the barrier to auditors trying to learn which pieces of regulation
 83   apply and when they apply.
 842. As you can imagine, the audit process for a firm of KPMG's size can be
 85   complex at times and changes depending on the client, the client's industry,
 86   the type of engagement, and more. Clara AI helps break down each of these
 87   challenges and provide detailed, accurate guidance for auditors to learn
 88   about their specific use-case without getting lost in irrelevant data.
 89
 90Clara AI does a lot more than I've written about here and I know teams are using
 91these features beyond what my teams do in Tech Assurance. You can read about
 92more features, such as automation, dynamic dashboards, and more on the product
 93page: [[https://kpmg.com/us/en/capabilities-services/audit-services/kpmg-clara.html][Clara AI]].
 94
 95* Use Cases for Each Tool
 96
 97While this section won't be exhaustive, it provides a few basic examples of what
 98my teams do at KPMG with AI, as well as some suggestions for how other auditors
 99can utilize AI at their own companies.
100
101** Gemini
102*** Client and Project-Based Agents or Notebooks
103
104As I mentioned above, one great use-case is to build notebooks for your client
105team or even for internal teams, such as a team that's formed to increase
106automation within the audit practice.
107
108I build notebooks for these teams, upload as many relevant documents and links
109as I can, invite my team members, and then save useful artifacts within the
110notebook so that my team members don't need to recreate the wheel in order to
111gain access to that same information.
112
113*** Technical Research
114
115Through purely anecdotal experiences, I have found Gemini to be superior to my
116other AI tools when it comes to technical topics. It provides more accurate
117information and better context for my specific role.
118
119This has replaced the experience of walking team members through topics, such as
120Snowflake architecture, AWS logical security, or SQL permissions.
121
122In addition, it helps me and others who are more experience dive deeper into
123subjects that we historically did not have the time or knowledge to research.
124For example, Gemini recently helped me use my knowledge of similar systems to
125understand Kafka's architecture and risks as they related to my client's
126specific implementation of Kafka.
127
128*** Data Analysis
129
130As I mentioned above, Gemini seems to excel with technical tasks. I've been
131using this to enhance the data analysis we perform.
132
133For example, Gemini is able to help write queries (e.g., SQL or AWS commands) to
134extract the information we need to test a control. While we have lots of
135internal documentation for popular platforms, there are always situations where
136you need data from a system and don't have any guidance to extract it.
137
138For users with data, Gemini is able to provide a more natural language querying
139process where the user can simply describe what analysis they need, or even
140generate a dashboard for the user. I find this most helpful in identifying
141relevant metrics or analytics and then have our teams use their skills to build
142out Alteryx workflows and Power BI dashboards to implement what Gemini has
143helped them investigate.
144
145Lastly, data cleaning is wonderful in Gemini. Gemini is able to help structure,
146clean and deduplicate data much faster than humans can and seems to have a lower
147potential for errors than if someone manually combed through the data trying to
148correct it.
149
150** Microsoft Copilot
151*** Meeting Transcripts
152
153One of the built-in features of Copilot in Teams is [[https://support.microsoft.com/en-us/office/view-live-transcription-in-microsoft-teams-meetings-dc1a8f23-2e20-4684-885e-2152e06a4a8b][transcription]]. This is a
154killer feature for auditors, as one of the biggest pain points for an audit (on
155both sides) is having to meet multiple times to rehash the same topic or send
156constant follow-ups throughout the year to re-confirm facts.
157
158With a transcript, we can spend one meeting going fully through all questions
159and simply refer back to the transcript whenever we have a question.
160
161If you also save the video recording, you can also save the time of requesting
162evidence later, as you already have a recording where you walked through the
163evidence you needed.
164
165This saves:
166
1671. An absolute *ton* of time on both the audit and client sides, in terms of
168   questions and follow-ups.
1692. Even *more* time by not needing the ask the client to take a million
170   screenshots. Instead, the auditor can take screenshots from the recording
171   after the fact. No more stopping every 5 seconds in a meeting waiting for
172   someone to take a screenshot or sending a request for screenshots after the
173   fact.
174
175*** Meeting Facilitator
176
177Another killer feature of Copilot is the [[https://learn.microsoft.com/en-us/microsoftteams/facilitator-teams][Facilitator]]. Facilitator takes the
178agenda in the meeting invite (or determines the agenda throughout the meeting),
179tracks progress of the discussion against the agenda, takes notes, and tracks
180to-do items with assignments for users to take action after the end of the
181meeting.
182
183*** Microsoft Office Integrations
184
185I talked about the Microsoft Office integrations above, so I won't dive deep in
186this section.
187
188For auditors, some useful examples are:
189
190- Generating boilerplate for an audience you're not familiar with writing for
191  (e.g., engineers vs. CFOs). Note that you *shouldn't* generate the content,
192  you should only let Copilot guide you to the proper format and then use your
193  knowledge to write the content.
194- Checking your tone, especially if Copilot has access to historical chats, in
195  order to help you respond professionally and appropriately.
196- In Excel or Word, helping you analyze your data.
197
198I'm not a big fan of generating content with Copilot, so I don't have any
199examples for that. Perhaps I am naive, but I still think we should be honing our
200interpersonal skills and the content of our communications with others should
201still be written by us, not AI.
202
203*** Managing Emails, Chats, and More
204
205I almost titled this section "Manager Mode", since I'm feeling a different level
206of intensity from the sheer number of emails, Teams messages, tags in channels,
207and more that I experience now as a Manager.
208
209One small but effective use for Copilot is to check at the end of the day if
210you've missed anything that day, such as responding to an email or message,
211actions items assigned to you in meetings, or important announcements that,
212while not directed at you individually, may require action from you.
213
214I currently do not have access to Copilot Studio, but I'd love to build a custom
215agent around this idea if I could.
216
217** Clara AI
218*** Firm & Methodology Guidance
219
220For an auditor who has to deal with SOX, SOC 1-3, AUP, and more engagements, you
221can imagine the sheer number of potential pitfalls there are when working
222through an engagement.
223
224Being able to have Clara AI break down the regulatory requirements for your
225specific situation is amazing.
226
227For example, how do you risk assess Client ABC, who has unique IT systems and
228business processes, and operates in the oil & gas industry? What if they are
229also impacted by the US's tariffs? Even further, what if they experienced a data
230breach in the prior year and have experienced changes in numerous key roles?
231
232Clara AI is able to handle of this and more, guiding team members to the
233relevant guidance and assisting them in the audit process.
234
235*** Workflow Assistance
236
237On top of the various regulations and internal guidance frameworks, auditors at
238the Big 4 also have to learn how to use their firm's specific audit platforms.
239
240KPMG built the KPMG Clara Workflow (KCw) for audits and SOC reports, which
241requires teams to fill out the audit file for their engagement, answer all
242questions that appear, perform their testing, and abide by the required items.
243
244Clara AI helps by connecting auditors to the internal guidance that supports
245each section of the audit platform so that they can complete their engagement in
246compliance with their requirements.
247
248* A Day with AI
249
250Before I finish, I wanted to provide an example of how an auditor might utilize
251AI throughout their day. You can probably summarize this already from the
252information provided above, but I wanted to lay out in plain terms here at the
253end:
254
255- Strategic planning in the morning with any AI. Review regulatory news,
256  summarize key decisions made by the government, industry leaders, and more.
257- If you have a lot of emails and messages, use Copilot to summarize actions
258  items and make a prioritized plan of action.
259- As you test or review items in the audit file, use Clara AI to identify gaps
260  in approach.
261- During meetings, enable transcriptions and Facilitator. Use these tools so
262  that people attending the meeting don't need to take detailed notes. Instead,
263  each person can be fully invested in the topics they're discussing.
264- Use Copilot to summarize and draft outlines for emails to the team and/or
265  client.
266
267** Conclusion
268*** What /Can/ AI Do?
269
270AI /can/ increase efficiency, consistency, and audit quality, but there are some
271big assumptions when we state that. We first need to make sure we, as auditors,
272are using the tools themselves effectively. For an auditor, a general AI chat
273won't be very useful. Instead, we should be training and using focused models
274that solve specific purposes. The model will perform better if it's trained for
275specific purpose and isn't expected to know everything that's ever existed.
276
277But what happens when we /do/ use AI effectively? Well, at that point, we are
278able to elevate ourselves and automate the manual and tedious tasks that have
279taken up time historically.
280
281Staff who may have been asked to spends hours cleaning data from the client
282before can now spend a shorter length of time reviewing the data output instead
283and then using that additional time to enhance their critical thinking. If you
284are used to performing these manual tasks, think about what your seniors or
285managers do with your results. Can you take on those tasks instead and elevate
286yourself into those roles?
287
288*** The Role of the Human in an AI World
289
290Last, but not least, I want to *highly stress* the importance of human in the
291loop (HITL) concept. What we do as auditors is strategically important, whether
292you're an internal or external auditor, and cannot be replaced by the current
293landscape of AI tools. Therefore, when we're using AI, we need to make sure we
294validate *everything* coming out the tool. At no point can the AI take
295responsibility, so we need to take responsibility for our actions when we use
296AI.
297
298However, when we do all of this, we are able to elevate the role of the auditor
299to one of judgment, critical thinking, validation, and client relationship
300management. This means faster, higher quality, and more impactful audits
301throughout the industry.