AI for Compliance Automation - Compliance AI Platform

Disclaimer: All data, records, policies, users, security events, compliance findings, and other information presented in this project are entirely fictitious and have been created solely for demonstration and experimentation purposes. No real organizational data, proprietary information, or personally identifiable information (PII) is included. This project is intended to showcase the design and implementation of an AI-powered compliance platform and should not be interpreted as representing the security posture, policies, or operations of any real organization. 

Ensuring that an organization is safe from cyberthreats requires strong policy and compliance guidelines and employees who are knowledgeable on these standards. Guaranteeing that all of the people and parts within the organization are complying is a challenging task that can be remediated with artificial intelligence. A large language model (LLM) can be particularly helpful with answering questions and compiling data. To combat the challenges of compliance security, I have put together a Compliance AI Platform that can perform a variety of compliance tasks that can assist users with their inquiries and provide clearer transparency among access and auditing.The platform uses Retrieval-Augmented Generation (RAG) to help the LLM find the specific data that it needs.This can significantly reduce the amount of time needed for a user to get the understanding that they need when searching for specific information. With the power of AI, an organization can significantly enhance its security through better access to compliance and policy data using the Compliance AI Platform.

The application has ten pages in total. The home page provides a broad overview of compliance metrics, recent assessments, security alerts, and key findings. It acts as a synopsis, giving the user a rundown of the ongoing security and compliance posture.

---

The next page is Policy QA, which assists with retrieving policy data. Users can ask the system a policy question and receive a verified response.

---

The Control Testing page is responsible for evaluating whether a security or compliance control is running successfully. When a control ID is provided, a status of pass/fail/warning assessment, evidence, and a recommendation are provided.

---

The Compliance Data page provides the user with a place to look over the evidence that backs the compliance activities, specifically with access reviews and policy exceptions. These tables inform the user of details such as who has access to what, what exceptions are allowed for policy requirements, and what has been approved.

---

The Audit Log page keeps a record of access information within the system. The log records who accessed system resources, when the access occurred, and what actions were performed. The AI Workspace page provides users with a chatbot for specific queries and requests, such as questions about policies or to create compliance assessments. RAG is utilized to assist the LLM in finding responses.

---

The Assessment History page lists the results of previously completed AI control assessments. The tables details when the evaluation happened, what was specifically examined, and what the outcome was.

---

The Evidence Explorer page allows users to browse and investigate compliance evidence that the organization has gathered. The page displays all of these records in a table and allows the users to scroll through each record to observe more details.

---

The System Health page presents how well the Compliance AI platform itself is performing. To create a clear understanding of how effectively the platform is running, the page displays the status of the database connection, data pipeline, vector store, and AI model availability.

---

The Report Generator page allows the user to create a report summarizing the organization’s compliance readiness, detailing the compliance findings, evidence, and AI assessments in a consolidated compliance report.

---

The architecture of the Compliance Evidence Pipeline is split into three major components: frontend, backend, and AI layer. The frontend sets up the user interaction for the tool. Streamlit dashboard is used to create the interface that allows the users to view and interact with the compliance evidence pipeline through its many pages. To access the platform, the user must sign into the interface with a username and password. The level of access that they have is determined by a specific role. The interactive reporting also assists the user in gaining insight on the organization’s compliance findings. The backend is responsible for storing and managing the data that is displayed in the tool. PostgreSQL is used for database storage to record data such as users, hosts, access reviews, and policy exceptions. SQLAlchemy models are utilized to transform this data so that it can work with Python. To capture the policy information, the data pipeline uses text files that go into detail on the specific policies and chunks, embeds, and stores the vectors so that specific responses can be retrieved. This moves us to the AI layer. Once the policy information has gone through this process, the RAG system can parse the vector storage for the specific response that it needs. The large language model that it is utilized (ChatGPT in this case), receives the request and finds the specific policy areas where it can locate the correct response. These three components working together allows for a smooth user experience when gathering data on compliance policy.

The RAG pipeline is what gives the Compliance AI Platform its AI abilities and makes searching for and putting together policy and compliance data easier. This is possible thanks to RAG’s ability to ingest the policy documents, embed the information, and retrieve this information when needed. Using langchain, the chunking breaks down the data into smaller pieces, creating a smoother process for the LLM when it is gathering the information that it needs for its responses. Metadata is also included to ensure that the source that the AI got the data from is not lost. These chunks are stored using a Chroma vector database, stashed in smaller pieces so that they can be easier found if needed by the LLM. When a user prompts the LLM with a request, the request is embedded and turned into a numerical vector. Chroma seeks through the vector database to correlate the vector to each chunk that is stowed. Once Chroma has found applicable chunks, a prompt is set up that contains the user’s request, the located chunks, and any instructions for how the model should respond. This prompt is sent to the LLM so that it can create a response that is sent back to the user. Through this method, RAG is able to simplify and improve the process of how an LLM responds to requests, allowing for better and faster compliance visibility within an organization.

Artificial intelligence can assist cybersecurity in several ways, and one significant approach to this is utilizing large language models and their ability to respond to specific requests and generate specific data insights. This project demonstrated the incredible potential for the use of AI within cybersecurity compliance visibility and policy comprehension. Through the use of retrieval-augmented generation, AI processes can be enhanced through its assistance to the LLM’s ability to get the data chunks that it needs to respond to specific requests. This ability opens the door for a variety of helpful applications from compliance reviews to policy querying to auditing logging. New advances in AI conveys new opportunities to improve cybersecurity processes through more than just automation, but through establishing a full platform that security professionals can rely on.

Code for project is available here: https://github.com/jack-gage-sec/cyber-ai-rag/tree/compliance-ai-platform

Cybersecurity AI Agents - Red and Blue Team

The emergence of agentic artificial intelligence in cybersecurity has been a game changer, especially with the use of large language models.These new abilities can perform both offensive and defensive actions, allowing for improved automation for both red team and blue team work. With help from Claude, I have created and successfully tested two AI cybersecurity agents with one acting as the attacker and the other acting as the defender.

The Purpose

Thanks to the abilities provided by LLMs like Claude, attacks can be orchestrated autonomously by agents, making it easier for hackers to target vulnerable systems. While these developments can invite the risk of bad actors who intend to hack for malicious purposes, this also provides security researchers with an easier way to find vulnerabilities in a system. Anthropic’s new development, Mythos, is an example of an LLM that uses its ability to find weaknesses within a structure. While the automated nature of these tools can make cybersecurity processes quicker, easier, and cheaper in many ways, it is important to recognize that human involvement in this process remains highly important. When conducting vulnerability assessments, there needs to be real people making judgment calls about how the agents are to go about their investigations. When creating the agents, I made sure to include a prompt that would both inform the user of what it was planning to do and require direct permission from the user before moving forward with its plans. With this safeguard, we ensure that the agent is not making serious decisions on its own with the explicit consent of a human being.

Architecture

Using a Kali Linux virtual machine as the offensive station and a Windows 10 virtual machine as the defensive station, I was ready to test both the red team agent and the blue team agent. I installed Damn Vulnerable Web App (DVWA) on the defensive station to serve as the vulnerable guinea pig. With help from Claude, the red team agent engages in reconning, planning, and vulnerability tests. Utilizing recon tools such as nmap and gobuster, the agent could gather information on the potential weaknesses of the defensive station. Once the agent has collected this, it can make plans on how it wants to test the vulnerabilities of the DVWA. When the plans are all set to go, the agent displays what it wants to do to the user and prompts them for permission to move forward with these plans. If the user responds with a “yes”, the agent begins with a series of penetration testing, such as SQL injection, brute force attacks, and command injections. After this is performed, the red team agent concludes running by providing the user with a report detailing which methods were successful in its vulnerability testing and a summary of how vulnerable that the target system is. Moving over to the defensive station, the blue team agent is at work searching for the offensive actions that were committed by the red team agent. Using the access log from the web server, this agent is able to analyze any interactions made with the website and determine if there have been any attacks performed and what address these attacks came from. The agent then concludes by providing the user with a report detailing what kind of attacks were performed and what defensive actions can be taken. Like the blue team agent, the red team agent utilizes Claude in its analysis.

Human Approval

An important element that I included in the red team agent was human insight being required before the vulnerability testing can take place. With Claude being used to analyze the situation and decide how to perform vulnerability testing, the agent has the ability to be fully autonomous. Allowing an agent with offensive cybersecurity abilities to run without human supervision invites serious risk into the situation. Even though this agent is designed to be used by ethical hackers that are supposed to test for vulnerabilities, there could be serious consequences if the will of the agent permits it to perform an action that was not considered by its human designers. This is why it is crucial for a person to both review the plan that the agent developed with help from Claude and for the person to give approval before the plan is enacted. A potential disastrous cybersecurity incident can be prevented with a simple “no” response to the agent’s prompt.

Red and Blue Tag Team

I chose to create both the red team agent on the offensive station and the blue team agent on the defensive station because I recognize the importance of both understanding how a system can be compromised and how these threats can be detected and prevented. The two agents act with significant independence from each other, with only the log file acting as a shared state. In a real-world scenario, the blue team agent would need to detect threats coming from places that it would have no previous involvement with, so it is crucial that its detection abilities are able to be performed separately from the red team agent.

Results

Running the agents on the virtual machines achieved interesting results. I intentionally set the security of the DVWA to a low setting. When performing recon, the red team agent accesses several possible vulnerabilities and detailed priority levels for which vulnerabilities could be best exploited. A plan was put together for how it was going to test its offensive maneuvers on the DVWA. I gave my approval, and the agent took its time launching its vulnerability testing. Once this was completed, the agent provided a detailed report on its success with its specific tests where it explained what was vulnerable and what was not. The red team agent had concluded its work, so I moved to the defensive VM where I ran the blue team agent. Using the access log, the agent provided a report stating that an attack took place, what was involved in the attack, where the attack came from, the most critical concerns to be considered, and what security measures should be taken The blue team agent detected all five methods of attack used by the red team agent, demonstrating its ability to recognize specific vulnerability breaches.

Limitations

While I had success on the tests that I ran, there are nonetheless limitations to the agents. I was testing it on Damn Vulnerable Web App specifically on the lowest security setting, so the bar is already set pretty low for the red team agent to perform its recon and vulnerability testing. I am interested to see if I am able to successfully pentest the app when the security setting is higher, as well as in different testing environments that are harder to crack than DVWA. Another element about the agents that I would change is increasing data minimization. The agents are passing tool outputs to Claude mostly unredacted so that Claude can directly analyze the raw findings to create reports and enhance detection in ways where a fixed signature could miss. For the virtual machine lab setting that I have set up, this is not a problem. However, if I were to utilize these agents in a production setting, there would need to be more significant limitations on what could get sent to Claude. This could be solved by using a local LLM or by filtering what gets sent to Claude to only include a vague description. When taking these options, there will need to be consideration in determining if these changes will reduce Claude’s effectiveness in making decisions for the agents. All of this can be researched in depth in the future as I continue to improve the abilities of the red and blue team agents.

What this demonstrates

The use of large language models in autonomous AI cybersecurity agents opens the door for improvement for both identifying vulnerabilities and detecting potential threats. With the power that this ability brings, it is crucial that there is still human involvement to ensure that the choices that the agents make are being reviewed and approved of so that serious damage can be prevented. With this cautionary measure included, the agents are able to work together to improve cybersecurity measures from both a red team and blue team perspective.

Code for both agents is available here: https://github.com/jack-gage-sec/cyber-ai-agent/tree/red-and-blue-team

Academic Work - Reinforcement Learning

• DQN Agents

  The document demonstrates reinforcement learning through the use of deep Q-network (DQN) agents by displaying how the agent an be used for both the Mountain Cart and Lunar Lander environments. The agent receives rewards and penalties for its actions as a way to teach the agent to make the most desirable decisions.

• Agentic RL

  This document demonstrates establishing a breakout environment and training a proximal policy optimization agent to how to behave within the environment.

Academic Work - Computer Vision

• Digital Image Processing and Image Preprocessing

  The two documents below demonstrate how an image can be digitally processed and preprocessed through the use of image modifying techniques and the image processing library OpenCV. The work performed in the first document features the altering of the Lena image, performing actions such as resizing, cropping, color manipulation, and binarization. The second document focuses on an image of a human blood vessel and making the image clearer through image augmentations, histogram stretching, CLAHE, RGB normalization, and brightness and contrast enhancement.

• Morphological Image Processing

  This document demonstrates image processing through using the geometry within the images. Through boundary extraction, closing, region filling, and connected components, morphological processing can help create an image that is easier to understand.

• Object Detection

  This document demonstrates Haar cascade classifying using blood vessel analysis and license plate reading. The blood vessels are dilated, closed, and have connected components identified. The license plate reading process uses erosion, edge detection, contour detection, and masking to determine the license plate number.

• YOLO

  This document demonstrates the use of YOLO (You Only Look Once) where filter boxes are used to identify specific objects in an image and a video.

Academic Work - Ethics Papers

The following annotated bibliographies demonstrate the significance of ethics in artificial intelligence through research into journals and articles. These papers cover artificial intelligence’s societal impact, current legal and ethical conversations, and implementation across several industries.