Akamai Engineering Culture
Akamai is an environment fueled by the desire to learn and improve. There are open engineering and training courses, wikis, live training sessions, as well as engineer lead lecture series. Most importantly there is a strong culture around continued personal and professional development.
The Security Operations Control Center (SOCC), for example, has a continuous training program where team members are given a full day, every week, dedicated to skills development. These sessions range from video training on new and old products, to lectures and presentations from fellow team members, engineers, and researchers from other departments.
The Akamai SIRT has been tasked to develop a more challenging, and ultimately entertaining, training program. This program gives SOCC engineers the opportunity to learn new skills and get highly technical hands on experience. The training - which is a CTF or Capture The Flag contest - is designed to be both amusing and challenging, offering the SOCC engineers the opportunity to use the tools they normally have at their disposal to tackle challenges mirroring attacks our customers face in a safe environment.
What is a CTF?
A CTF is a popular competition among hackers. Participants are tasked with solving sets of challenges from different categories of technical know-how. For every challenge they solve, they're awarded points.
Most importantly, CTFs require participants to be creative, use critical thinking, and problem solving before ultimately arriving at a functional solution. These challenges will likely be more difficult than anything they're likely to encounter in the real world, and it won't be something they're going to be able to find the solution to on Google (although they'll certainly be using Google... a lot). They'll have to put in the work, understand the process, and only then will they be rewarded with their answer, their points, and their rightful position on the leaderboard.
There are two main styles of CTF: Red vs. Blue and Jeopardy style. Red vs. Blue format CTF is where a Red Team group is tasked with attacking servers and resources on a shared network, while the Blue Team is expected to defend those same systems, detect intrusions, and keep the environment secure. Jeopardy style CTFs maintain a scoreboard which tracks progress on questions and categories, awarding points for solves. Questions and categories are offered with different point values where the higher the points awarded for a solve, the more difficult the challenge should be to complete.
How is this used to train Engineers?
Akamai SIRT team members often compete online and offline in CTF competitions, sometimes individually, sometimes as a unified team. As a form of training and practice many members of the SIRT team swear by CTFs as their greatest training asset. The quality of this experience and honing of skills is truly hard to beat.
As part of their ongoing training at Akamai, the SOCC participates in a CTF built by the Akamai SIRT, often referred to as a Capture the Packet (CTP) competition. This is a Jeopardy style CTF that focuses on capturing, analyzing, and carving network traffic from live taps (on an isolated network). How they must handle that traffic is dictated by the challenges and the required information needed to submit their solve. This particular format was chosen, as all SOCC Specialists must know network packet analysis, as it's an absolute necessity in their professional role within the SOCC.
Participants inspect live network traffic that is replayed at randomized times, for the time-limited duration of the game session, in order to answer questions on a Jeopardy-style scoreboard. This allows the participants to fine-tune their skill in carving out specific network traffic samples to analyze. Questions usually have multiple layers or hurdles. The first hurdle in this style of competition being able to isolate and carve out the needed traffic from the network. Other obstacles could include rebuilding files from the network traffic, obfuscation, steganography, cryptography, and usually some level of automation/scripting. The challenges and time limits help develop critical thinking, problem solving, and help with team building.
All of this sounds great in theory, but what makes it even more enticing is the price tag. These sessions are typically organized by groups of hackers and volunteers, are available for anyone to participate in, and in nearly all cases, are absolutely free (except for a couple pizzas and some energy drinks you might need).
Building a CTF to train your Engineers?
Competing in a CTF is a wonderful way to challenge your engineers and their skills, but building out a CTF is easier said than done. It often demands resources in the form of time, technical knowledge, and creativity to create and operate.
While there are plenty of open source options for CTF scoreboards and systems that can be used, a majority of the time you'll spend building the CTF will go into designing and creating the challenges themselves. It's like designing highly technical puzzles for hackers. You can't just make something that is impossible for anyone to solve.
Organizers must make sure the puzzles are solvable, yet challenging enough for the participants to engage in creative problem-solving. The CTF puzzles should cover an array of skill-sets and skill-levels too. Lower point challenges might require identifying a bad actor exfiltrating files over a clear-text protocol such as FTP, more advanced challenges may require a player to reconstruct a zip file from network traffic, and brute force its password, before deobfuscating a text file. Some of the most advanced challenges may require reverse engineering custom built binaries, or require crafting an exploit payload that is required to break into a remote machine on the competition network.
Once the challenges have been devised, built, and are proven solvable, organizers must set the game in motion and begin seeding the challenges within network traffic to create packet captures that are replayable across a purpose built competition network - being sure to inject plenty of noise and useless traffic for the seeds to hide amongst.
While this sounds straightforward, unexpected hurdles can occur. When developing the first iterations of the SIRT CTP challenges, it was discovered that some of them were unsolvable when tested on the physical CTP network. This was due to the packet lengths of some captured traffic being too large, which made the challenge captures unable to be replayed across the competition network. After quite a bit of digging, they discovered that this was due to the traffic being captured on a server that we did not disable segmentation offloading.
This is just one of the gremlins that can be hiding in your well laid plans, so be sure to run some test sessions before the big day, or you may be in for a very stressful real-time debugging session with a room full of bored engineers. Although this could also be a fantastic learning experience!
Having a CTF'ing good time
The value of CTF competitions as a training and learning aide can't be stressed enough. However, going into your first CTF can be a bit overwhelming. The odds that you'll quickly encounter problems that require specialized tools, knowledge, and skills is almost guaranteed. This is why it's important to be prepared for battle.
"In Washington I met a lumberjack who said that if he were given five minutes to chop down a tree or lose his life if he failed to do so, he'd spend three of the five minutes sharpening his axe." - Darrell Royal
From a skills perspective, participants should have some experience with automation via scripting, programming, and/or command line piping. It's advised to choose and practice your scripting language of choice, but while trying to decide which one, you should consider the wealth of tools available in the standard and open source libraries available.
Some practical languages for these types of challenges include Bash, Golang, and Python. Python has a wealth of existing examples, libraries, and frameworks (such as pwntools) that make it fast to develop tools capable of turning daunting tasks into a few lines of code and a solution.
Arguably the second most important skill will be ones Goog-fu. Knowing the proper way to ask the right question is an invaluable skill. It's very likely reading about tools, how to use them, combing over docs, stackoverflow posts, and github will all be linchpins in the quest to completing a given puzzle. The fewer cycles spent chasing the wrong answers the better.
Having a low level understanding of underlying computing concepts is yet another invaluable skill. Knowledge about topics such as networking, protocols, binary conversion and how to handle/process it, packet sniffing, obfuscation/cryptography techniques, and common exploitation techniques (i.e. OWASP Top 10) will all be useful. More advanced skill sets that are used in the advanced challenges could include binary reversing, static analysis, exploitation development, Linux and Windows internals, and advanced cryptography.
Last but certainly not least, participants are strongly encouraged to get comfortable with Linux. The great news is, thanks to powerful laptops available and freely available virtualization technology stacks, this has never been easier. In the case of a machine for CTF uses, most distros will work just fine. But there are a few that are significantly better than the rest due to them being purpose built and equipped with a wide selection of tools pre-installed. Distros such as Kali, Parrot, BackBox, Samurai, ArchStrike, BlackArch, Pentoo, and many others can be found online, typically with a VM image that can be downloaded and run in your hypervisor of choice.
If all of this sounds fun and like a challenge that you're ready to get into, then the final piece is finding your first CTF! The great news is there are plenty to pick from. There are a couple different types online. Traditional CTFs, which are typically time-limited events with live participants and a score board, but you can also practice/enjoy Crackmes and other open long running CTFs.
Sites like ctftime.org will be a hub to find ongoing and upcoming events that an individual or team can participate in. Additionally there are several groups that maintain their own sets of challenges, places like ringzer0ctf.com, hackthissite.org, crackmes.one, and many others can be found with a quick search. It's also good to read write-ups put together by teams after CTFs have ended, they serve as a sort of bragging rights, where they provide work throughs of how they managed to solve some challenges they faced while competing github.com/ctfs.
If you're looking for a team of peers to learn with and try your hand in a group setting with some very friendly and very talented CTF'ers take a look at the OpenToAll CTF team. They're a great bunch of folks with an emphasis on learning and competing.
Feedback from SOCC Engineers
Following the training sessions the SIRT will review the questions and let the engineers explain their solves as a group, additionally unsolved challenges are reviewed and explained in-depth. Many engineers were happy to provide feedback when asked how they felt about the training exercises, below are unedited feedback from some of the participants.
"Overall the CTF was a great training exercise. It was helpful to have experience performing deep dive analysis on traffic we don't normally see, looking for data points and anomalies we aren't normally concerned with. This presented us with a great opportunity to learn tools and techniques we otherwise wouldn't develop. It was also a good troubleshooting exercise to have to initialize the packet capture and ensure that it was consumable by our tools. In general the CTF seemed adequate in depth and difficulty for the time constraint, and it may also be beneficial to review once all teams have finished competing."
Security Operations Specialist
"Overall they enjoyed the opportunity as it is definitely out of the normal routine training and as you mentioned, promotes creativity, critical thinking and even teamwork. It also gives technicians and less advanced Specialists the opportunity to learn and build on their technical skills (or lack thereof) vs watching training videos. Hands on training such as the CTF engages the individuals more, keeps them interested and can definitely retain more information this way."
We hope you'll consider adding CTF style training programs to your organization and engineering teams curriculum. At Akamai we've found them to be invaluable as a training aid, and a great way to encourage your teams and peers to challenge themselves and broaden their skills in a fun, interactive, and meaningful way.
One last thing...
All this talk of CTF'ing and challenges is great and all, but how about we give you a simple challenge from the SIRTs own CTP challenges to try for yourself? This was a 100 point challenge in the obfuscation category, and should be quickly solvable by a seasoned veteran.
An email was sent with a encoded message. What is the flag? The email traffic carved out. After following the traffic stream you see. Can you decode the message and get the flag?
220 kali ESMTP Exim 4.91 Thu, 13 Dec 2018 10:10:32 -0500
250-kali Hello localhost [127.0.0.1]
mail FROM:<Jestated65@jourrapide.com> size=751
354 Enter message, ending with "." on a line by itself
Content-Type: multipart/mixed; boundary="===============3164830132405445185=="
Subject: Email Data - Test email
Content-Type: text/plain; charset="us-ascii"
250 OK id=1gXSdE-0001cm-JK
421 kali lost input connection