Commit 51c534c9b9
Unsigned
Layout: unified · split
content/blog/2025-11-23-it-audit-ai.org added +300
| @@ -0,0 +1,300 @@ | |||
| 1 | #+date: <2025-11-23 17:43:57> | ||
| 2 | #+title: Practical Uses for AI in IT Audit | ||
| 3 | #+description: Learn how I'm currently using AI in my role in Technology Assurance (i.e., IT Audit) at KPMG. | ||
| 4 | #+slug: it-audit-ai | ||
| 5 | |||
| 6 | * My AI Toolkit | ||
| 7 | |||
| 8 | As others in the Big 4 will probably tell you, AI is a strategic tool that | ||
| 9 | allows for a far more efficient and impact audit. However, the key is | ||
| 10 | understanding when to use it, how to use it, and how to translate the output | ||
| 11 | from your AI into actions. | ||
| 12 | |||
| 13 | This post will walk you through how I personally leverage AI in my role. | ||
| 14 | |||
| 15 | Let's start by stepping through the tools I have at my disposal. | ||
| 16 | |||
| 17 | ** Gemini Enterprise (formerly Agentspace) | ||
| 18 | |||
| 19 | First and foremost is [[https://cloud.google.com/gemini-enterprise][Gemini Enterprise]], which has a large context window, is | ||
| 20 | connected to the Enterprise's internal documentation and websites, and allows | ||
| 21 | for specialized use of custom agents, [[https://notebooklm.google/][NotebookLM]], and more. | ||
| 22 | |||
| 23 | I use Gemini for a handful of specific use-cases, notably: | ||
| 24 | |||
| 25 | 1. NotebookLM notebooks for team-based usage. For example, you can build a | ||
| 26 | notebook for a specific client (using public and private information about | ||
| 27 | that client) and then chat with the data in the notebook, generate an audio | ||
| 28 | overview, and save documents, such as mind maps or study guides, for future | ||
| 29 | use. | ||
| 30 | - I love this because it provides a "run once and save" functionality, where | ||
| 31 | I can generate things like an audio overview or document, save it, and then | ||
| 32 | my team can reference it without re-running the prompt or even needing to | ||
| 33 | open the notebook, if they download it. | ||
| 34 | 2. Custom Agents for customized personas that benefit from the private | ||
| 35 | information that Gemini has access to, such as internal SharePoint sites. | ||
| 36 | - For example, I have a custom agent designed to operate as an IT Audit | ||
| 37 | Assistant that can help you ask questions that can be answered by the | ||
| 38 | company's news, documentation sites, and more (e.g., "I'm testing logical | ||
| 39 | access for AWS and want to know if anyone has created a solution or | ||
| 40 | guidance for XYZ."). | ||
| 41 | |||
| 42 | ** Microsoft Copilot | ||
| 43 | |||
| 44 | Likely the most common AI in use at company's today, due to its integration with | ||
| 45 | every Microsoft product, Copilot offers the best AI solution for productivity. | ||
| 46 | However, it's usage is likely also the most volatile due to the large gap in | ||
| 47 | skill sets between new-hires and senior employees. | ||
| 48 | |||
| 49 | For associates and other early-career employees, I suggest that they use Copilot | ||
| 50 | to perform two primary tasks: | ||
| 51 | |||
| 52 | 1. Check their tone, such as making an email or messages more professionalism, | ||
| 53 | softer, or perhaps to analyze the conversation up to that point to understand | ||
| 54 | how to respond better. This is something that /should/ come naturally to | ||
| 55 | senior employees, but is often a learning curve for associates. | ||
| 56 | 2. For written documentation, such as guides or presentations, use Copilot and | ||
| 57 | provide it with ample information on the goal of the documentation in order | ||
| 58 | for Copilot to suggest an outline, gaps in the writer's thinking, or areas | ||
| 59 | for further research before writing the document. The key here is to use | ||
| 60 | Copilot to improve your thinking and not to replace it. It shouldn't generate | ||
| 61 | the content for you, it should remove the tedium of creating boilerplate | ||
| 62 | structures or to design the structure of a document. | ||
| 63 | |||
| 64 | ** Clara AI | ||
| 65 | |||
| 66 | [[https://kpmg.com/us/en/capabilities-services/audit-services/kpmg-clara.html][Clara AI]] is KPMG's audit-based AI platform, known as the "KPMG Clara Smart Audit | ||
| 67 | Platform". The major benefit to Clara AI is that it lives within a sandboxed | ||
| 68 | environment, running on white-labelled models, fine-tuned on firm-specific | ||
| 69 | methodologies, and built directly into KPMG's audit tools. | ||
| 70 | |||
| 71 | As noted in KPMG's Clara AI page linked above, Clara AI is built directly into | ||
| 72 | the Clara audit platform in order to provide systematic risk assessments, drive | ||
| 73 | consistent and efficient audits by providing all audit teams with relevant | ||
| 74 | information for their audit, and connects to other tools (such as KPMG Client | ||
| 75 | Collaboration, where evidence is requested and provided). | ||
| 76 | |||
| 77 | I *absolutely love this tool*, as it solves some of the bigger problems for | ||
| 78 | auditors at large firms: | ||
| 79 | |||
| 80 | 1. Connecting younger staff to the firm's guidance and regulatory material, | ||
| 81 | lowering the barrier to auditors trying to learn which pieces of regulation | ||
| 82 | apply and when they apply. | ||
| 83 | 2. As you can imagine, the audit process for a firm of KPMG's size can be | ||
| 84 | complex at times and changes depending on the client, the client's industry, | ||
| 85 | the type of engagement, and more. Clara AI helps break down each of these | ||
| 86 | challenges and provide detailed, accurate guidance for auditors to learn | ||
| 87 | about their specific use-case without getting lost in irrelevant data. | ||
| 88 | |||
| 89 | Clara AI does a lot more than I've written about here and I know teams are using | ||
| 90 | these features beyond what my teams do in Tech Assurance. You can read about | ||
| 91 | more features, such as automation, dynamic dashboards, and more on the product | ||
| 92 | page: [[https://kpmg.com/us/en/capabilities-services/audit-services/kpmg-clara.html][Clara AI]]. | ||
| 93 | |||
| 94 | * Use Cases for Each Tool | ||
| 95 | |||
| 96 | While this section won't be exhaustive, it provides a few basic examples of what | ||
| 97 | my teams do at KPMG with AI, as well as some suggestions for how other auditors | ||
| 98 | can utilize AI at their own companies. | ||
| 99 | |||
| 100 | ** Gemini | ||
| 101 | *** Client and Project-Based Agents or Notebooks | ||
| 102 | |||
| 103 | As I mentioned above, one great use-case is to build notebooks for your client | ||
| 104 | team or even for internal teams, such as a team that's formed to increase | ||
| 105 | automation within the audit practice. | ||
| 106 | |||
| 107 | I build notebooks for these teams, upload as many relevant documents and links | ||
| 108 | as I can, invite my team members, and then save useful artifacts within the | ||
| 109 | notebook so that my team members don't need to recreate the wheel in order to | ||
| 110 | gain access to that same information. | ||
| 111 | |||
| 112 | *** Technical Research | ||
| 113 | |||
| 114 | Through purely anecdotal experiences, I have found Gemini to be superior to my | ||
| 115 | other AI tools when it comes to technical topics. It provides more accurate | ||
| 116 | information and better context for my specific role. | ||
| 117 | |||
| 118 | This has replaced the experience of walking team members through topics, such as | ||
| 119 | Snowflake architecture, AWS logical security, or SQL permissions. | ||
| 120 | |||
| 121 | In addition, it helps me and others who are more experience dive deeper into | ||
| 122 | subjects that we historically did not have the time or knowledge to research. | ||
| 123 | For example, Gemini recently helped me use my knowledge of similar systems to | ||
| 124 | understand Kafka's architecture and risks as they related to my client's | ||
| 125 | specific implementation of Kafka. | ||
| 126 | |||
| 127 | *** Data Analysis | ||
| 128 | |||
| 129 | As I mentioned above, Gemini seems to excel with technical tasks. I've been | ||
| 130 | using this to enhance the data analysis we perform. | ||
| 131 | |||
| 132 | For example, Gemini is able to help write queries (e.g., SQL or AWS commands) to | ||
| 133 | extract the information we need to test a control. While we have lots of | ||
| 134 | internal documentation for popular platforms, there are always situations where | ||
| 135 | you need data from a system and don't have any guidance to extract it. | ||
| 136 | |||
| 137 | For users with data, Gemini is able to provide a more natural language querying | ||
| 138 | process where the user can simply describe what analysis they need, or even | ||
| 139 | generate a dashboard for the user. I find this most helpful in identifying | ||
| 140 | relevant metrics or analytics and then have our teams use their skills to build | ||
| 141 | out Alteryx workflows and Power BI dashboards to implement what Gemini has | ||
| 142 | helped them investigate. | ||
| 143 | |||
| 144 | Lastly, data cleaning is wonderful in Gemini. Gemini is able to help structure, | ||
| 145 | clean and deduplicate data much faster than humans can and seems to have a lower | ||
| 146 | potential for errors than if someone manually combed through the data trying to | ||
| 147 | correct it. | ||
| 148 | |||
| 149 | ** Microsoft Copilot | ||
| 150 | *** Meeting Transcripts | ||
| 151 | |||
| 152 | One of the built-in features of Copilot in Teams is [[https://support.microsoft.com/en-us/office/view-live-transcription-in-microsoft-teams-meetings-dc1a8f23-2e20-4684-885e-2152e06a4a8b][transcription]]. This is a | ||
| 153 | killer feature for auditors, as one of the biggest pain points for an audit (on | ||
| 154 | both sides) is having to meet multiple times to rehash the same topic or send | ||
| 155 | constant follow-ups throughout the year to re-confirm facts. | ||
| 156 | |||
| 157 | With a transcript, we can spend one meeting going fully through all questions | ||
| 158 | and simply refer back to the transcript whenever we have a question. | ||
| 159 | |||
| 160 | If you also save the video recording, you can also save the time of requesting | ||
| 161 | evidence later, as you already have a recording where you walked through the | ||
| 162 | evidence you needed. | ||
| 163 | |||
| 164 | This saves: | ||
| 165 | |||
| 166 | 1. An absolute *ton* of time on both the audit and client sides, in terms of | ||
| 167 | questions and follow-ups. | ||
| 168 | 2. Even *more* time by not needing the ask the client to take a million | ||
| 169 | screenshots. Instead, the auditor can take screenshots from the recording | ||
| 170 | after the fact. No more stopping every 5 seconds in a meeting waiting for | ||
| 171 | someone to take a screenshot or sending a request for screenshots after the | ||
| 172 | fact. | ||
| 173 | |||
| 174 | *** Meeting Facilitator | ||
| 175 | |||
| 176 | Another killer feature of Copilot is the [[https://learn.microsoft.com/en-us/microsoftteams/facilitator-teams][Facilitator]]. Facilitator takes the | ||
| 177 | agenda in the meeting invite (or determines the agenda throughout the meeting), | ||
| 178 | tracks progress of the discussion against the agenda, takes notes, and tracks | ||
| 179 | to-do items with assignments for users to take action after the end of the | ||
| 180 | meeting. | ||
| 181 | |||
| 182 | *** Microsoft Office Integrations | ||
| 183 | |||
| 184 | I talked about the Microsoft Office integrations above, so I won't dive deep in | ||
| 185 | this section. | ||
| 186 | |||
| 187 | For auditors, some useful examples are: | ||
| 188 | |||
| 189 | - Generating boilerplate for an audience you're not familiar with writing for | ||
| 190 | (e.g., engineers vs. CFOs). Note that you *shouldn't* generate the content, | ||
| 191 | you should only let Copilot guide you to the proper format and then use your | ||
| 192 | knowledge to write the content. | ||
| 193 | - Checking your tone, especially if Copilot has access to historical chats, in | ||
| 194 | order to help you respond professionally and appropriately. | ||
| 195 | - In Excel or Word, helping you analyze your data. | ||
| 196 | |||
| 197 | I'm not a big fan of generating content with Copilot, so I don't have any | ||
| 198 | examples for that. Perhaps I am naive, but I still think we should be honing our | ||
| 199 | interpersonal skills and the content of our communications with others should | ||
| 200 | still be written by us, not AI. | ||
| 201 | |||
| 202 | *** Managing Emails, Chats, and More | ||
| 203 | |||
| 204 | I almost titled this section "Manager Mode", since I'm feeling a different level | ||
| 205 | of intensity from the sheer number of emails, Teams messages, tags in channels, | ||
| 206 | and more that I experience now as a Manager. | ||
| 207 | |||
| 208 | One small but effective use for Copilot is to check at the end of the day if | ||
| 209 | you've missed anything that day, such as responding to an email or message, | ||
| 210 | actions items assigned to you in meetings, or important announcements that, | ||
| 211 | while not directed at you individually, may require action from you. | ||
| 212 | |||
| 213 | I currently do not have access to Copilot Studio, but I'd love to build a custom | ||
| 214 | agent around this idea if I could. | ||
| 215 | |||
| 216 | ** Clara AI | ||
| 217 | *** Firm & Methodology Guidance | ||
| 218 | |||
| 219 | For an auditor who has to deal with SOX, SOC 1-3, AUP, and more engagements, you | ||
| 220 | can imagine the sheer number of potential pitfalls there are when working | ||
| 221 | through an engagement. | ||
| 222 | |||
| 223 | Being able to have Clara AI break down the regulatory requirements for your | ||
| 224 | specific situation is amazing. | ||
| 225 | |||
| 226 | For example, how do you risk assess Client ABC, who has unique IT systems and | ||
| 227 | business processes, and operates in the oil & gas industry? What if they are | ||
| 228 | also impacted by the US's tariffs? Even further, what if they experienced a data | ||
| 229 | breach in the prior year and have experienced changes in numerous key roles? | ||
| 230 | |||
| 231 | Clara AI is able to handle of this and more, guiding team members to the | ||
| 232 | relevant guidance and assisting them in the audit process. | ||
| 233 | |||
| 234 | *** Workflow Assistance | ||
| 235 | |||
| 236 | On top of the various regulations and internal guidance frameworks, auditors at | ||
| 237 | the Big 4 also have to learn how to use their firm's specific audit platforms. | ||
| 238 | |||
| 239 | KPMG built the KPMG Clara Workflow (KCw) for audits and SOC reports, which | ||
| 240 | requires teams to fill out the audit file for their engagement, answer all | ||
| 241 | questions that appear, perform their testing, and abide by the required items. | ||
| 242 | |||
| 243 | Clara AI helps by connecting auditors to the internal guidance that supports | ||
| 244 | each section of the audit platform so that they can complete their engagement in | ||
| 245 | compliance with their requirements. | ||
| 246 | |||
| 247 | * A Day with AI | ||
| 248 | |||
| 249 | Before I finish, I wanted to provide an example of how an auditor might utilize | ||
| 250 | AI throughout their day. You can probably summarize this already from the | ||
| 251 | information provided above, but I wanted to lay out in plain terms here at the | ||
| 252 | end: | ||
| 253 | |||
| 254 | - Strategic planning in the morning with any AI. Review regulatory news, | ||
| 255 | summarize key decisions made by the government, industry leaders, and more. | ||
| 256 | - If you have a lot of emails and messages, use Copilot to summarize actions | ||
| 257 | items and make a prioritized plan of action. | ||
| 258 | - As you test or review items in the audit file, use Clara AI to identify gaps | ||
| 259 | in approach. | ||
| 260 | - During meetings, enable transcriptions and Facilitator. Use these tools so | ||
| 261 | that people attending the meeting don't need to take detailed notes. Instead, | ||
| 262 | each person can be fully invested in the topics they're discussing. | ||
| 263 | - Use Copilot to summarize and draft outlines for emails to the team and/or | ||
| 264 | client. | ||
| 265 | |||
| 266 | ** Conclusion | ||
| 267 | *** What /Can/ AI Do? | ||
| 268 | |||
| 269 | AI /can/ increase efficiency, consistency, and audit quality, but there are some | ||
| 270 | big assumptions when we state that. We first need to make sure we, as auditors, | ||
| 271 | are using the tools themselves effectively. For an auditor, a general AI chat | ||
| 272 | won't be very useful. Instead, we should be training and using focused models | ||
| 273 | that solve specific purposes. The model will perform better if it's trained for | ||
| 274 | specific purpose and isn't expected to know everything that's ever existed. | ||
| 275 | |||
| 276 | But what happens when we /do/ use AI effectively? Well, at that point, we are | ||
| 277 | able to elevate ourselves and automate the manual and tedious tasks that have | ||
| 278 | taken up time historically. | ||
| 279 | |||
| 280 | Staff who may have been asked to spends hours cleaning data from the client | ||
| 281 | before can now spend a shorter length of time reviewing the data output instead | ||
| 282 | and then using that additional time to enhance their critical thinking. If you | ||
| 283 | are used to performing these manual tasks, think about what your seniors or | ||
| 284 | managers do with your results. Can you take on those tasks instead and elevate | ||
| 285 | yourself into those roles? | ||
| 286 | |||
| 287 | *** The Role of the Human in an AI World | ||
| 288 | |||
| 289 | Last, but not least, I want to *highly stress* the importance of human in the | ||
| 290 | loop (HITL) concept. What we do as auditors is strategically important, whether | ||
| 291 | you're an internal or external auditor, and cannot be replaced by the current | ||
| 292 | landscape of AI tools. Therefore, when we're using AI, we need to make sure we | ||
| 293 | validate *everything* coming out the tool. At no point can the AI take | ||
| 294 | responsibility, so we need to take responsibility for our actions when we use | ||
| 295 | AI. | ||
| 296 | |||
| 297 | However, when we do all of this, we are able to elevate the role of the auditor | ||
| 298 | to one of judgment, critical thinking, validation, and client relationship | ||
| 299 | management. This means faster, higher quality, and more impactful audits | ||
| 300 | throughout the industry. | ||