My Server Audit Checklist
I perform regular audits on my servers to analyze ssh access logs, user account audits, installed software reports, and installed dependency reprots. These reports are used in vulnerability scanning software and gives us the ability to act in an agile manner when something goes wrong.
One of the tasks that I do on a regular basis is my server audit checklist. This gives me insights into which software is installed on the systems, the system users, the bill of materials for the software I write and deploy, and the users that have SSH access. Allowing me to manage the servers, review vulnerability reports, and manage and limit access efficiently as a solo developer. This information combined with tools can make any developer insanely productive and efficient.
Locally Installed Software
The first thing I like to know about my systems are the software directly installed on the Operating System. This is a very improtant task that can be dealt with by only installing the software that is absolutely necessary for my software to operate. This reduces the amount of software that I need to worry about vulnerabilities in. This task can be performed by using Operating Systems like NixOS or TalOS depending on the environment you are installing in. Systems like NixOS and TalOS allow you to update all of your servers with a single git commit. I won't go into how this all works in this article, as it is a deep topic.
Pairing those immutable Linux Operating Systems with a competent package audit process can give you all of the insights you need to keep your system secure in a single developer environment. We categorize the package names and the package versions so we can keep track of vulnerabilities and behave in an agile manner when patching up our servers in production.
User Access
The second thing we keep track of is the users that can access the systems. We want to audit and monitor the system users, services users, and automation users thouroughly to ensure that nobody has gained access to the servers that should not have it as well as keep an audit trail that shows us the user activity on each account. We will go over each item that we audit on these users.
User File Access
The first thing we audit about the users is the files that they have permission to. I want to make sure that I know every file that an individual user has ownership of as well as that user's permission on each individual file. This starts to paint a picture of which services that particular hacker is targeting.
User Commands
The next thing we audit about the users is the commands they are executing on the system. Often times the file system audit will give us a good idea of what commands they will/have run against the system, but we can figure out more about their intent by capturing all user commands and logging them in a central location.
Software Bill of Materials
It is not unheard of for malicious users to poison python and node dependencies to gain access to servers or exfiltrate your database credentials or even the database itself.
Software Bill of Materials Audit
The bill of materials audit is where all software package audits start. We catalog every package we use in our software. This gives us a list of package names and the versions that we can perform our software package audit on. When performing software bill-of-materials audits, you will still need to understand what to do when a vulnerability is found.
Software Package Audit
Auditing the software packages you already use is a great way to determine if your software is currently affected by a vulnerability or a bug. This gives you the insights required to mitigate the vulnerability or migrate to a version that is not affected by this issue. While I cannot go into great depth on how to spot these vulnerabilities as that will depend on the frameworks and libraries used to develop the software. We use a lot of white hat and black hat techniques on internally hosted and completely on-premises servers to deliver software that is as secure as we can make it. Remember not to perform these attacks in hosted environments in the cloud as you don't own the hardware or the software running in the cloud and can face legal scrutiny and even face prison time.
SSH Access
The final, but probably the most important thing to audit is the list of users that can access the machine via SSH.
SSH Access Audit Trail
It's really important that we keep track of a few specific config settings and the users that can remote in using the SSH protocol. The settings we want to keep tabs on are the AllowPasswordAuthentication setting and the PermitRootLogin setting. Both of these settings should be turned off to force users to require a private key to access the server AND ensure that nobody can log in as the root user on the system. This greatly simplifies securing your system and keeping tabs on it regularly via automated tools and reports.
Fail2Ban Jail Audit
Fail2Ban is a really helpful tool that bans ip addresses from accessing your server when a malicious user attempts to gain access. It is helpful to keep track of where these users are based on the geographical information attached to their ip addesses. By taking the jailed user's ip addresses and collecting the request data on the backend, we can trace back a malicious user to their geographic location. Potentially multiple hops back to the originating user's location in more sophisticated attacks.
Takeaways
The takeaway that I want to emphasize here is that the first step to great security and user privacy is limiting what you have to secure and then making the software components that are on your system observable. By reducing the number of methods that an attacker can gain access to a system, you will reduce the work to defend against these attacks. By regularly performing these audits and the tasks attributed to them, you will make it easier to capture these bad actors and shut them out of your systems. Just remember that your system cannot be perfect all of the time. But we can get very close to it most of the time.