Qualifiers | Building DEF CON CTF

This is part 3 of a series of posts about Building DEF CON Capture the Flag.

Quals, to me, is the most important part of DEF CON CTF: it’s the only game we make that most teams will have any interaction with, and for the teams that do qualify, it’s the best way to prepare them and give them an idea about what the finals are going to be like. We come to a consensus about dates in December, and try to have date announcements out on January 1. Yes, we opened quals registration on April 1 each year on purpose.

Picking April or May for qualifiers is important, for several reasons. 2013, our first year running it, we only knew we were hosting by March, so we picked mid-June like previous organizers. This only gave us a month and a half for finals prep, which just felt like a panic. More importantly, this only gave players traveling from some countries six weeks to go through the US visa process.

Actually building qualifiers is a lot of work! Challenges area a whole post or series of posts on their own, but the important parts are brainstorming ideas for them, making sure they’re solvable by teams, and getting them running stably on infrastructure that will survive the game. Challenges need to be tested by someone coming in with a blank slate on it, and the challenge author needs to make something that can solve it reliably in production.

Estimating difficulty is hard work, so what we found works best is just guessing at an unlock order a few hours before the game starts, and scoring challenges based on how many times they’re solved.

Production operations for challenges is worth thinking about at development time. Challenges as stdio binaries that don’t save any state to disk between connections are good. We built runc images that would be launched by xinetd, and that consistently worked great.

The scoreboard isn’t terribly difficult. There’re lots of Jeopardy-style CTF scoreboards available, running a web application is a turnkey thing, and make sure the database gets backed up (we backed up hourly and before deploys.) Let teams register during the competition and make a public scoreboard (for non-logged-in visitors) available and obvious. Lots of players don’t think to register until the game is actually on, and many players want a link to share how they’re doing with friends, family or coworkers.

We found a lot of value in being in the same place for qualifiers. In 2013 and 2014, we used an office a bunch of the team worked from. 2015 and 2016 we used somebody’s house (and didn’t even trash it). 2017 we rented a party house for the weekend, and it was mostly good. We spent a lot of time in the pool, nobody had to drive home on a heavy drinking night, and it was a great time! Unfortunately, the internet was slow and LTE was jank, but we worked around that.

Qualifiers isn’t an easy game to run per se, but it’s very rewarding, and the lack of constraints for preparation meant that it was usually a pretty smooth experience for us.

Coming Soon:

  • Building Finals

Thanks Matthew Pancia for proofreading and reviewing.

Hype, Meetings, and Workflows | Building DEF CON CTF

This is part 2 of a series of posts about Building DEF CON Capture the Flag.

Basic Hype Game

Many parties want to see you succeed running DEF CON CTF: the DEF CON organization, past organizers, past competitors, and everyone in the CTF community. We all have a vested interest in seeing DEF CON CTF as a popular game with lots of players, which means you need to bring your hype game.

I consider our marketing/hype efforts under way once we launch the website, usually on Jan. 1. For us, this meant agreeing on quals dates in December. Dates are mostly arbitrary, but Jan. 1 was a convenient deadline to target, and gets the rest of the team in a CTF frame of mind.

Besides the website, letting people know about the upcoming game is useful. We kept a Twitter account active, posting announcements leading up to and during the game. Mentioning DEF CON’s official account is easy to do incidentally, and they’ll retweet CTF stuff to their zillion followers. We also had public Google Plus and Facebook pages, but I never felt like they got the traffic that Twitter did.

A CTF game is inherently fun and easy to advertise to CTF enthusiasts. What we’ve always struggled with is non-CTF enthusiast/professional types, especially after establishing a reputation as being very binary-heavy. We leaned in to it a bit, hyping up a challenge or two in 2017 as being web-based when they were just binary reversing that happened to speak HTTP.

One thing I enjoyed and think helped us was having the “#” (octothorpe) brand. The vine-covered computer was a good recognizable image in 2014, and the spraypaint-style version from 2015 has stuck around since then.

Meetings

In 2017, we met every Wednesday from January until Vegas. We didn’t go all Robert’s Rules of Order but I did maintain meeting notes each week, and persisted any ongoing stuff that needed/expected work from week to week to make sure it was getting done.

Keeping meetings on track is hard! Start with an agenda and know which items are likely to become an open-ended discussion (for us it was challenge infrastructure and challenge difficulty). Make sure that when open-ended discussions come up, you interrupt and defer them until the end of the call. You’ll either forget the less-exciting stuff, or not give it the attention it needs.

Keep the meetings friendly and fun! You’re all relying on each other, and if meetings go badly or turn personal, you’ll find tasks will slow down or never get done.

Workflows

We didn’t use Scrum™ or any other documented workflow. I kept a personal board on Trello of stuff to do, but didn’t expect anyone else to use it.

Be ruthless about things that don’t need to get done right away or ever. A full month after 2017 qualifiers, when I hadn’t finished the quals stats dump, Gyno told me to let it slide until after finals (I swear I’ll get it done one of these weeks). Ambition can be good: that’s why you’re running a CTF in the first place, and it’s where legendary challenges come from. But it’s risky, and when you have a deadline, sometimes you just want something you know you can do reliably.

Coming Soon:

  • Building Qualifiers
  • Building Finals

Thanks Matthew Pancia for proofreading and reviewing.

cLEMENCy - Showing Mercy

With the closing of DEF CON 25 and our last year of running Capture The Flag, I figured a post about what it took to create cLEMENCy was in order. This is a very long write-up detailing what I went through while developing cLEMENCy and ignores all the effort that occurred on top of it to create other challenges for DEF CON CTF 24 and 25. Hopefully this post shows the amount of dedication it can take to run DEF CON CTF. When I joined Legitimate Business Syndicate in January 2014, I made it known to the team that for our last year I wanted to do a fully custom architecture. As luck would have it, the Cyber Grand Challenge (CGC) happened in 2016 which allowed me to take a backseat to most of the CTF challenges and focus my evenings on the emulator and tool development.

My first processor document was created on August 11, 2014. Highlights were:

  • Stack growing upwards
  • Little Endian
  • 25 instruction groups
  • 32 registers that could be used as integer or floating point
  • 8 interrupts
  • DMA transfer involving the ID of a device you want to talk to in a register along with all areas that were potentially relevant
  • Memory protections
There were questions about whether or not I should add in logic to allow threading and if the firmware was static or a custom format that allowed modules. A month later, Gyno dropped in notes about Hexagon DSP, Mill, and the Cell architecture. I had tossed around other ideas in my head in relation to Harvard architectures and the idea of swapping out the opcode lookup table as opcodes execute, tying a unique table to each opcode resulting in code obfuscation.

2015

I began laying out the basics of the opcodes and started writing actual emulator code at the end of November 2015, almost a full 2 years after joining the team while pondering design ideas during that time. The development started shortly after my break from CTF finals that year and after a number of rapid updates and changes to the architecture idea document. Although the original architecture document called it “Lightning CPU” I officially named the project “DMC” (Defcon Middle-endian Computer) in December, I own a Delorean and wanted to tie in a reference to the Delorean Motor Company.

A large amount of development on the emulator occurred at the end of December due to having a 2 week vacation. By the end of December, 23 files, 1,867 lines of code, and 344 lines of comments had been put together for the emulator. This obviously does not count refactoring and various reworking that occurred at the early stages, however a total of 2,211 lines in around 30 days was not bad (almost 74 lines a day). Things were adjusted and more refined for the specification during this time while Thing2 and I chatted about middle-endian. Ideas were still floating around about replicating the SPARC sliding register window and allowing registers to combine to expand the number of bits used for math.

I created a personal goal of keeping things simplistic enough to allow teams to learn the architecture in less than a week with the idea of just handing the teams a manual a week before the competition. The end of 2015 was spent with random ideas and tweaks being done to align it to be RISC-like and refining the running document to remove complexities to help tailor the architecture to a setup that should be easy and quick to learn.

In order for the team to be able to develop challenges I needed to have a full toolset for them to work with. Work was started in January 2016 to create a LLVM configuration for the clang compiler after giving up on adapting GCC to middle endian. The complexity of modifying LLVM resulted in looking at a number of simple C compilers including TCC before stumbling across NeatCC (NCC) developed by Ali Gholami Rudi. The benefits of it were that the core file per architecture was simplistic, its author had a lightweight libc that compiled in it, along with a linker while supporting the ELF object format to link together multiple C files.

2016

The first few months of 2016 involved minor modifications to NeatCC to get it ready for creating firmwares for DMC along with creating a python script that would auto-generate a Sqlite3 database with information about opcode layouts. The database had been planned as it would allow auto-generating a header file of instruction data for a C disassembler that the emulator would eventually have. The database also would drive the planned python assembler and disassembler to avoid massive parsing code and also be used for creating the documentation.

By mid-April 2016, NCC, the Neat linker (NLD), and the emulator were usable. I also started testing between the tools and emulator. During this time period the python script to parse the Sqlite3 database and generate the initial HTML documentation was created along with the addition of more instructions. NCC did not have the ability to have embedded assembly so an external assembler was created, Lightning Assembler (LAS). June is when the architecture was re-named “cLEMENCy” (LEgitbs Middle ENdian Computer) and all code updated to reflect it. I wasn’t completely happy with the DMC name and enjoyed that clemency means mercy, which I was showing by not going all-out in complexity.

It was during DEF CON 24 CTF Finals that things changed. Thing2 and I were talking that Friday about how to break tools with the architecture when I had an epiphany! I asked him if making all bytes 9 bits would break everything, and he could only smile at the idea. I ran it past the rest of the LegitBS team and the consensus was that, if I could prove it would work, then why not? While all the competing teams worked on finals that year, I was creating a proof-of-concept. By the end of the competition, I was able to prove that I could not only convert between 8- and 9-bit formats easily, but that I could make it work with the tooling and setup I had previously developed.

By the end of September, the instruction format, assembler, disassembler, NLD, and NCC were converted over to the new 9-bit byte layout. In the process, I came across a number of parsing issues in NCC that were not showing up in the latest release. I made a new pull of NeatCC, NeatLD, and NeatLibc in the beginning of October, spending 2 weeks dealing with a massive merge and rewrite of the code to make it compatible with the new setup.

Mid-October 2016 was the beginning of modifications to the emulator to adapt it to the 27-bit, 9-bits-per-byte setup - another 2-week process. While doing the rewrite, debugger and disassembler functionality started to be added to the emulator allowing for simple and expected functionality, dumping registers, breakpoints, and single stepping to name a few.

The speed of the emulator was a bit slow, though, so instead of adding floating point logic to NCC, I chose to remove the floating point code. This eliminated the masking, compare, and branch on most of the opcodes while requiring that I add in functionality to indicate if the processor supported floating point to avoid documentation changes. This improved the speed of the emulator to around 8 million instructions/sec on my old laptop and I deemed it good enough as the infrastructure would be a bit beefier. It appeared that the biggest speed hit was the constant shifting and masking required to handle 9 bit byte access.

November 2016 saw modifications and assembly file additions to NeatLibc specific for cLEMENCy and the actual creation of full firmware images across the tools that the emulator would run outside of the simple assembly tests that had been done after the 9 bit conversion. I had to create a custom memory allocator in Neatlibc because the original one relied on mmap() and the firmware had fixed memory to work with. LAS was modified to be capable of writing ELF objects that were compatible so the assembly could be linked in.

The rest of November and December were spent squashing bugs, adding in enhancements, and writing assembly files for things like millisleep (similar to nanosleep). During this time the addition of code for inverting the stack by recompiling ncc to ncc-inv was also added. The fun of tracking bugs at this point was determining if the emulator, disassembler, assembler, linker, or compiler was the culprit. Some bugs were issues of improper masking in the emulator, some were the C compiler kicking out the wrong values due to improper bit combining, while others were from the linker incorrectly writing offsets or incorrectly calculating where to modify data. While trying to validate issues, bugs would be found in edge cases in the assembler and disassembler.

2017

By the middle January 2017 enough bugs had been squashed that Perplexity, my Finals challenge that I’m sad to say was never finished, was being used to test functionality fairly ruggedly. Perplexity’s goal was to make people question if I had created a C++ compiler for the architecture on top of everything else that was developed. It was a C binary with structures setup to allow vtable configurations while NCC was modified to allow two colons side by side in a function name so that everything looked like C++ in development. In January 2017 the plan was to release just the manual of the architecture to the teams although there was a debate of how early the teams should have the manual. There was also a question of whether or not all challenges would be written in it due to the LegitBS team not using any of the tools yet.

By the end of January 2017, the emulator, assembler, backend file for the C compiler, and custom files for NeatLibc totaled 54 files and 13k total lines of pure code not counting comments. Sirgoon had begun working on a physical version of the processor on a FPGA, however due to personal things this was scrapped. If anyone ever makes this into hardware I would love a copy of it.

The tools and emulator were stable enough in my testing that I had nothing to fix and just needed to write my finals challenge until others began using it and reporting bugs. Work progressed on Perplexity with random additions to the debugging abilities of cLEMENCy. At this point there was still no plan to release any of this to the teams outside of the manual that had not been modified since end of December 2016.

Come April, the DBRK instruction was added to help pinpoint parts of the code after being recompiled. Although the map file existed, I did not have line specific information so identifying specific areas after a recompile was faster after adding DBRK. This also allowed me to quickly test theories about ways to land Perplexity for some of the bugs I had already added.

We decided to rent a house for our last time running quals. During this time I continued working on Perplexity, and showing the team the tools was very helpful. Since we were face to face, they had faster access to questions and issues. It was also requested during this time that shared memory be added, and as a bonus I added the NVRAM memory. There were plans for a challenge to use the shared memory, but I’m not aware of it actually occurring. During quals we determined that one connection per team per service would limit the load on the boxes and no planned challenges required multiple connections. This limitation was implemented and tested during quals.

After quals, Vito put in effort to start getting a physical manual created, Selir worked on porting old challenges so we could benchmark how much processing power was required with all services being attacked, and the rest of the team began using the tools to create and port challenges they had been working on. This became a pressure point for me due to a bunch of random directions on the documentation, tools, and architecture and of course bugs began to show up.

I created a separate slack channel to track just emulator changes that the team could be aware of and people could report issues in. It was not uncommon for a single day to have 10+ random small changes and bug fixes done to the tools and emulator with the weekend being up to 20+ different things. As we moved closer to Finals, the rate of updates and bug fixes would increase.

Just to list a few random bugs fixed in one weekend as a taste of things that were fixed:

  • Some of the branch compares had improper checks resulting in invalid if statements
  • Memory protections on the flag area needed to be enforced
  • Millisleep and strcpy having edge cases to correct
  • Exiting debug in certain situations resulted in an unusable terminal
  • Timers not firing properly
  • Signed immediate issues
Normally when challenges for Finals were created, a number of bugs would be added to the service with proof of concepts showing at least gaining control of PC and ability to continue. Due to the new architecture I made a request that the team accepted and agreed on; any bug in a challenge must be proven to be able to return the flag. This resulted in the creation of a rop-search tool to help prove that bugs could be landed in the challenges and an issue was discovered, our limited code size and lack of threading meant it was near impossible to do a stack pivot if you only control a register and PC.

Gyno came up with the idea of a page of memory that would have gadgets in it. Vito and I were about ready to get the physical manuals printed but I had not figured out exact details for this new memory area. I decided to leave the memory area out of the official documentation with the reason that a physical processor wouldn’t have this memory page, it was just from the emulator and being in the DMA area could be seen as a separate device. I created a script to auto-generate a random character text version of the LegitBS logo combined with text to appear similar to a NFO from a warez leak. The concept was to leave a few hints: the help menu saying to enjoy the NFO section, and an ELF section named NFO with the normal ASCII text. Reading it would tell them where the NFO was loaded in memory and the random character setup was to mask embedded ROP gadgets for pivoting from any register to stack.

Near the end of June 2017 we decided that the teams would be given a copy of the emulator along with the built-in debugger and disassembler, and the architecture manual 24 hours in advance. We decided this after watching lunixbochs (on the usercorn team) implement the NDH CPU into an emulator in a few hours. By releasing the emulator and built-in tools it would guarantee at minimum that everyone had the tools required to compete and avoided time being wasted at the start of the game to get tools created. I had not planned on teams having the emulator so Perplexity development was shelved. At this point Perplexity was 43 files and 2,808 lines of code, not counting comments. I was sad to shelve it but I needed to make sure the emulator was ready for teams. This involved me creating 3 versions of the emulator when it compiled, the production version with seccomp and stripped out debugging, our debug build, and the team debug build that had the instruction and register state history stripped out.

Near the end of the last week prior to Finals things appeared to be going smoothly. The emulator was chugging away at 7M instructions/sec on my laptop and we were not overloading our infrastructure that was being tested with multiple sample binaries. Selir had configured random connections and data for the sample binaries between all fake teams to help stress test and watch CPU spikes. I made a painful discovery; I found that I made a mistake in a select statement.

Can you spot the mistype? A simple 1 to 0 resulted in my poor laptop cranking out 40M instructions/sec easily, an almost 6x speed up. The speed boost was nice but I had to keep from kicking myself over tossing floating-point early in the process over such a simple mistype. We were close enough to game day that adding floating point back in was risky and wasn't needed as challenges had already been written to work around and avoid it's usage.

Game Day

I arrived to Vegas on Tuesday, July 25. Previous years I always had a challenge in finals and was doing last minute changes up till game start, adding in bugs, testing functionality, and enhancing the poller script. This year was different; I had no challenge, just the architecture. The team was spread out between multiple hotel rooms and a lot of time I was just sitting around. I decided to toss my laptop in a backpack and just wander Caesars. The team knew that I had my phone and a ping on Hangouts or Slack with a room number would result in me showing up. I had only ever gone to DEF CON for CTF, which had been 6 years straight and always busy. I had competed for 3 years and helped run for 3 years already, and having my 7th year to just wander was odd and surreal. I wandered around while the team was busy finalizing their challenges but I had nothing to really accomplish until an issue was discovered or someone needed help. On Thursday we released everything and were alerted to a couple mistypes in the manual. The delay on an answer and fix was because I was off in the swag line and needed to get back to one of the rooms as I refused to use my WiFi. Thursday is also when a clang bug was discovered; using the -O3 optimization was actually causing an edge case in the networking code. After a couple hours of testing and watching the bug appear and disappear purely based on adding debug logic, I recompiled with -O2 and everything worked as it should have.

Friday, I hated this day with a passion. From everyone else’s perspective Friday kicked off well, challenges appeared to go off without issue, HITCON landed 3 different first bloods, everything appeared to be going like clockwork. Behind the scenes I couldn’t stop shaking before game start and refused to drink anything due to being a light-weight. I needed to be able to think straight if a last minute fix was needed due to things going to hell. We had done testing, we had a version of the emulator running that no teams had to help avoid a breakout if one existed, but if something broke it would likely be on my head, I was stressed.

Mid-day Friday a bug was discovered in the custom malloc, it wasn’t re-handing out blocks and once fixed, 2 high memory usage and unreleased services would randomly fault. Sirgoon and I spent all of Friday afternoon going through the allocator. By the end of Friday I was done and actually feeling sick, probably from the stress I put on myself. We had fixed one of the services but the other continued to act up for unknown reasons. Sirgoon and Selir saved me, Saturday morning I found out they stayed up after Friday’s competition and hand verified the allocator to each other and found no faults with it outside of what was fixed Friday afternoon. They then in turn started looking at the challenge and found a null deref. Because of the offset it would wrap to high memory to get the allocated block information. This high memory was Read/Write although writes were ignored resulting in no crashes but invalid information leaking into the memory block chain. The rest of the weekend was far more relaxing and a huge weight off my shoulders.

Closing

Back in 2014 I had two teams approach me at Finals swearing up and down we had a custom architecture that year before game start as they both knew my background. I pointed out the difficulty in creating something custom, the tooling and testing required and tried to give the idea that it was hard. When it was announced at closing ceremonies in 2016 that we were doing a custom architecture the room was quiet for a moment, it was eerie. A number of the top teams knew me as the one that created the hardest challenges and also knew that if a custom architecture was to be done I would be involved. Although teams did not know I was the sole creator of the architecture until shortly before the contest, I heard rumors that teams were afraid of what would be created due to my involvement. I’m actually proud that I struck fear into teams. Balancing ease of learning with complexity and something new is not easy. I spent a lot of time tweaking ideas and judging if I thought the teams would adapt well enough before beginning to write any actual code. I could have done far worse but am glad to see my years of effort paid off and that it was thoroughly enjoyed.

I have created a number of custom architectures during my years of development and I hope that my challenges and cLEMENCy will leave a mark on the CTF scene. I thoroughly enjoyed creating my masterpieces and also learned that some people of my team and the competing teams think I am insane. I am going to have Perplexity with all of my notes pushed to the LegitBS repo when finals challenges are pushed. It was never finished but shows what could have been. Thank you not only to Legitimate Business Syndicate for asking me to join the team but for also supporting my ideas and plans. I also have to thank each and every player that took on my challenges over the years even if they didn’t solve it. I hope I helped others strive to learn new things and push their knowledge limits.

-Lightning

Team Building and Proposing | Building DEF CON CTF

It’s been a great honor and pleasure to be part of Legitimate Business Syndicate while hosting DEF CON Capture the Flag for the last five years. Now that DEF CON has announced the selection process for the next DEF CON CTF organizers, it’s time for the next organizers to step up to the plate, take the reins, and mix their metaphors on the way to becoming the new DEF CON Capture the Flag organizers. I hope you’ll find this series of posts useful when building your next Capture the Flag game, or writing a proposal for the big one.

Team Building

The most important part of running DEF CON CTF is the team you run it with. You have to trust in your other teammates’ skills, because running a complex, multi-challenge CTF alone simply isn’t doable. Before you can organize a CTF, you first must organize a team. This team should have skills with network operations, application operations (devops), forward and reverse engineering of complex networked services, full-stack database-backed web application development, real-time computer graphics, visual design, and more. More than the skills though, team members should be able to explain and share their knowledge. If you’re not cross-training skills among your team, you’re actively harming yourself.

Make sure team members know what they're getting in to. Learn what DEF CON CTF means for them, why DEF CON CTF is special for them, and why they should dedicate years of their life to it.

Naming your group is important! We got a lot of mileage about all the permutations of “Legitimate Business Syndicate.”

You don’t have a team without a way of communicating with the rest of your team. When we started in 2012-2013, we used a private Google Plus group. Since the end of the 2013 CTF season, we’ve used Slack, which you’re almost certainly familiar with. We also have a ton of stuff on Google Drive: meeting notes, material for publication, expenses, etc.

We’ve shared the same gitolite install for the whole time. Infrastructure projects keep to one repo per project, like the ideal “Twelve Factor App.” Challenges tend to live in per-challenge-author repos, because challenges are pretty fluid and git merges can be hard. Lightning and I had a lot of just normal merge conflicts while I was working on the typesetting system for the cLEMENCy manual, and that was enough to knock both of us out of our workflows.

Writing the Proposal

We wrote our proposal basically as soon as we knew what questions we had to answer. Get everyone together in the same room for a whole weekend, a month before the response is due. Seriously. Submit that shit early. DEF CON will forgive some weirdness and inconsistency more than they'll forgive lateness.

Get everyone who’ll help you run the game your first year together and just hash it out over a weekend. Even if there’s a three hour drive involved. If there’s a flight involved. Not even joking. The proposal will be what guides you and advises you for the hardest CTF you’ve ever participated in.

We put 95% of ours together over a single weekend, a month before it was due.

Seriously, Just Fucking Do It, the only thing you have to lose is you not hosting DEF CON CTF.

Coming Soon:

  • Basic Hype Game
  • Meetings
  • Building Qualifiers
  • Building Finals

Thanks Lightning, Murmus, and Zap for proofreading and reviewing.