Why Worldr Took the Road Less Travelled: Early Security Testing

At Worldr, we set out to make security a core feature of what we do. We shifted security left and enabled both the developers and DevOps to take responsibility for security matters. This allowed us to have everything ready before the security audit, thus allowing us an opportunity to follow our own internal procedures for high standards in security.

All too often, security is an afterthought added once the code is nearly done and released. Sometimes it has an egg shell model also known as bee hive where once in, you are allowed to do anything. Or perhaps security policies are too restrictive and complex. In addition, few developers are trained to think in terms of infrastructure and security.

Before committing code changes, having a good set of git pre-commit hooks can save you a lot of time down the line by making the code clearer and more robust. Every pull request needs to be properly reviewed, to have tests to verify the code does what it is supposed to do, and have an appropriate security review. The latter does not have to take much time, nor is it appropriate for all changes. Nonetheless, it is essential to think about security at this stage.

Security does not have to be complicated and is better when simplified. Something like the threat model manifesto and the twelve factor app can give a solid foundation on patterns that work and anti-patterns to avoid.

For example, what happens when the database an application is using vanishes? Does the application just send errors to a log file which no one reads? Does the application die? Does it try to reconnect and when does it give up? If an Object-Relational Mapping (ORM) is used, what does it do in this case?

While limiting third party vendor modules is good, usability matters. The more dependencies, the higher the risk of potential security issues and updating them all is both risky and a long process. However, doing everything yourself is foolish. Not only will it take too long but you will introduce bugs of your own making without the benefit of many eyes looking at the code. Having a clear view of what external dependencies are in your system allows you to manage the risks better.

From the DevOps side, we based our infrastructure on trusted third party vendors with a strong reputation for security. We apply encryption in transit and encryption at rest to all services we run as well as transparent data encryption for our database. A great deal of automation has made things like releases trivial so that all relevant parts of the system are properly and easily updated.

The principle of least privilege access is present throughout our system. For example, the administrator of the  database server would be unable to access the database’s data since it is encrypted. Even a restart of the service would not help since the database cannot request its own decryption key: it needs to be provided by another service which alerts administrators every time it does so.

It means that both teams have visibility of what the other is doing and can support each other. A well-oiled machine indeed.

Once all the infrastructure and code were ready, we needed to find the right auditor. Someone who was both trustworthy and competent at breaking into systems. We asked Edgar Ter Danielyan who has an impressive resume spanning more than two decades in security at firms such as Microsoft, Deloitte and Royal Bank of Scotland.

We provided a basic machine with security measures applied according to our own advice from our user documentation. In a real deployment, this would be installed and secured by the customer. We also provide a minimal set of security hardening instructions which we strongly advise should be followed. The machine was set up according to those but nothing more. A tool like goss (quick and easy server validation) can help guide customers to make sure they have done this simple hardening.

Edgar accessed the machine with SSH so he had full visibility of the infrastructure and could experiment at will. We described the internals of the system to him so he had a good knowledge of it instead of just doing black box testing.

After a week of penetration and security tests, Edgar awarded us with a pass as no critical, severe, or major issues were found.

He raised a few issues however, all were minor and did not impact the overall security of the system but instead focused on accepted best practices. Of course, we fixed them all before asking Edgar to do a retest. After all, why have an audit if you do not follow its findings?

My thanks go to Dima Rozhdestvensky our Head of DevOps and to Michael Antipin our VP of Engineering for their hard work and leadership in ensuring that security is a top priority. And, of course, to Edgar Ter Danielyan for doing a thorough audit.

Yann holds a DPhil in theoretical astrophysics and has worked in applied mathematical research in Bayesian statistics and game, graph, and optimization theories. Yann specializes in performance critical algorithm implementation through to high-level application architectures all the way down to tiny scripts. He is the director of IT Security at Worldr Technologies.