DEF CON CTF 2016 is using the Cyber Grand Challenge Game Format

In 2016, DEF CON CTF will use the same game format as the DARPA Cyber Grand Challenge (CGC). It will not be the same attack-defense format it has been in previous years. The challenges will all be DECREE x86 binaries, instead of a wide sampling of challenge formats. Grab a beverage and some index cards, it's complicated.

What is the Cyber Grand Challenge game format?

The CGC game format is notionally similar to attack-defense. Teams receive vulnerable software, field patched versions, and launch exploits against other teams. The big differences are what aspects of the game teams control, and the role the scoring system plays in running the game.

We Hope You Like APIs

Because DEF CON CTF 2016 will feature the winning autonomous Cyber Reasoning System (CRS) from the Cyber Grand Challenge, our scoring system will provide the same APIs as CGC to all teams.

There will be a human-friendly interface that supports every feature of the CRS APIs. The CGC API is available today, and you can get a leg up on competitors by building tooling around it in advance. You did build baby’s first CRS for the thousand cuts quals challenges, right?

Virtual Competition is a CGC game API simulator that can be run locally. The virtual-competition source is also available.

Teams Don’t Control a Server

Attack-defense teams are usually given a privileged account (although maybe not root) on servers, which allows them to replace binaries, kill errant processes, and other activities.

In CGC-format games, teams don’t have privileged access to the servers that run their software; instead, they upload software as a DECREE Replacement Challenge Binary (RCB) directly to the scoring system. Finally, teams don’t launch their own exploits; instead, teams build exploits using one of the DECREE Proof-of-Vulnerability (PoV) formats, and upload them to the scoring system.

Understanding Proofs of Vulnerability is a high-level overview of how PoVs work. The sample challenge sets NRFIN_00073, CROMU_00070, and CROMU_00071 have working examples of PoVs that can be tested locally.

The virtual-competition system can accept and validate uploads of RCBs, firewall rules, and PoVs, but only stores them, and doesn't run them.

Teams Get a Programmable Firewall

Unlike previous Legitimate Business Syndicate games, defending teams have the option of writing firewall rules for the CGC network appliance. Just like RCBs and PoVs, these are uploaded to the scoring system.

Additionally, the firewall provides dumps of both poller and PoV traffic to teams over UDP.

Using the Network Appliance from the CGC documentation is a good starting point for how to make firewall rules. The network appliance implementation provided by DARPA is open-source. cb-packet-log is a tool for receiving packet dumps from the firewall.

Patched Challenges and Firewall Rules Are Shared

There’s very limited security-by-obscurity in commercial, industrial, and open-source software. Patches are analyzed to understand the vulnerability they fix, new software can be tested in lab conditions, third-party patches for major vulnerabilities aren’t unheard-of, and there’s little to be done about it.

Replacement challenge binaries and firewall rules will be shared among teams as they’re fielded. This means you can figure out how PPP’s patching progressed, or just field their binary yourself #yolo

How Do We Play It?

Just like DEF CON CTF years past, the game is divided into rounds. During each round, the functionality of teams’ challenges is evaluated, exploits fire, teams upload new stuff, and scores change. There is a bit of lag, and that’s where things get interesting.

Game Start

At game start, teams will be able to grab a list of currently-enabled challenge sets, and download the challenge binaries for them. As the game progresses, more challenge sets will be come available, and teams will download fresh binaries for them.

Analysis

The challenge binaries are DECREE executables. Analyze them how you wish.

In addition to the CBs, traffic tested against binaries fielded by your team is sent to you. This includes both poller and exploit traffic.

‘Sploits

In round 16, team “ShadyTel” has got an exploit that they’d like to field against “Milliways.” They create a C-based PoV for the exploit, and upload it the the scoring system. It’s an unreliable exploit, so they want to run it ten times per round.

curl --digest -u shadytel:loud \
  -F throws=10 \
  -F csid=LEGIT_00006 \
  -F [email protected]_milliways \
  -F team=2 \
  https://ti.defcon.legitbs.net/pov

Starting in round 17, the scoring system runs the ShadyTel PoV against the Milliways challenge set ten times, mixed in with regular poller traffic. If it negotiates and successfully proves a register control (Type 1) or private memory disclosure (Type 2) vulnerability, ShadyTel gets offense points and Milliways loses them.

Patching

Milliways sees the ShadyTel PoV traffic (it ran ten times a round, super noisy), confirm that they’re vulnerable to it, and produce a patched Replacement Challenge Binary. They upload it in round 20.

curl --digest -u milliways:dome \
  -F csid=LEGIT_00006 \
  -F [email protected] \
  https://ti.defcon.legitbs.net/rcb

In round 21, Milliways fails all their pollers by design. Patching has a cost. Any PoVs against it automatically fail too. Milliways' new RCB also becomes available to other teams. ShadyTel downloads it. So does team “Psychoholics.”

curl -f --digest -u shadytel:loud \
 https://ti.defcon.legitbs.net/round/21/evaluation/cb/2

In round 22, Milliways’ new RCB passes pollers.

Scoring

During their analysis, the Psychoholics notice that their in-progress replacement has already fixed the vulnerability Milliways patched, but that Milliways also tuned up some code to require less space, use less memory, and run faster. They gank it for their patch, since in addition to being graded on security and availability, teams are graded on efficiency as well.

curl -f --digest -u psychoholics:pdp11 \
  https://ti.defcon.legitbs.net/status

But You Don’t Have To Take My Word For It

CGC is an unprecedented investment in the future of CTF, and while it feels extremely academic today, we're confident that CGC-derived technologies are going to be a fixture in CTF from now on. We’re extremely excited to see what the future of CTF will be like, and we think you will be too.

Quals Wrapup

Greetings,

DEF CON 2016 CTF Qualifiers are officially over. Thanks to everybody who came by our IRC this weekend and played in our game. HUGE props to PPP, who solved every challenge available with just under 6 hours left in the game.

Source code to all of our challenges from this year is already up and posted on our github.

In addition to source code, the challenges, as they ran, are all available on Docker Hub. If you want to run a challenge from this year, it's as easy as `docker run -it legitbs/challengename`. Our docker hub page is https://hub.docker.com/r/legitbs.

In the coming days, we will be contacting the team captains of all the teams who qualified to confirm participation in finals. Please be on the look out for this email.

As a reminder: this year, DEF CON CTF Finals will be running on DECREE. We will be inviting the winning CRS from DARPA's Cyber Grand Challenge to compete against our qualified humans to see if they can stave off the Rise of the Machines. Details of the finer details of our game are forthcoming, be on the lookout for that.

From all of us at Legitimate Business Syndicate, we thank you for letting us run your Capture The Flag. We hope everybody had a good time playing, and we look forward to seeing all of you at the Bally's Event Center in Las Vegas!

DEF CON CTF Qualifiers for 2016 Starting Soon

Hey, we're running online qualifiers for DEF CON CTF at midnight UTC (five hours from now), and you should play them.

Really Obvious Foreshadowing

Quals this year have a lot of references to DARPA Cyber Grand Challenge technologies. If you have no idea what that is, read our CGC for Hackers series of posts. If you do have an idea of what that is but are fuzzy on some of the details, read those posts, and maybe also tear into our collection of CGC technical documentation. If you get bamboozled by all of the above, perhaps keep our CGC Glossary handy too.

• vito goes back to the flag mines…

What is DECREE?

DARPA’s Cyber Grand Challenge uses a new computing environment, the DARPA Experimental Cyber Research Evaluation Environment, or DECREE. While DECREE is designed to be less useful for general-purpose computing, it has some properties that make it excellent for binary reverse engineering, patching, and exploitation contests.

DECREE is built on i386 Linux, and ships with the clang compiler and support for special CGC Executable Format (CGCEF) binaries that support a much smaller set of system calls. In addition, DECREE comes with tools for validating functionality of binaries with XML-based pollers, proving vulnerabilities with XML- or C-based Proofs-of-Vulnerability (PoVs), and, of course, tools to help you debug and analyze binaries.

How do I install DECREE?

The current DECREE version is a small network of Vagrant boxes, available from the CGC repo. You'll need Vagrant and VirtualBox installed to get them running.

  1. Install VirtualBox from https://www.virtualbox.org
  2. Install Vagrant from https://www.vagrantup.com
  3. Download the Vagrantfile from http://repo.cybergrandchallenge.com/boxes/
  4. Put the Vagrantfile in a parent directory to where you’ll be keeping your DECREE files. For example:
    decree
        ├── Vagrantfile
        ├── example1
        │   ├── Makefile
        │   ├── lib
        │   ├── pov
        │   └── src
        └── example2
            ├── Makefile
            └── …
  5. Run vagrant up crs to download, start, and set up the image for cyber reasoning systems. This may take a while, so read about Vagrant at https://www.vagrantup.com/docs/getting-started/up.html while it works.
  6. Run vagrant ssh crs to ssh in to the image.
  7. The path with the Vagrantfile is at /vagrant, so cd /vagrant/example1
  8. Run make to build and test your CGC binary.

What is a CGC binary and how do I get one?

The crs Vagrant machine has challenge binaries in the /usr/share/cgc-sample/challenges directory.

Outside of the crs image, you can get samples at https://github.com/cybergrandchallenge/samples. The complete set of CGC Qualifying Event (CQE) challenge sets are there, as well as some example CGC Final Event (CFE) challenges in the “examples” directory.

What can I do and not do in a CGC binary?

CGC binaries are designed to be idempotent, like a pure mathematical function. Given the same initial conditions and inputs, the same binary should produce the same outputs, with the same instruction stream and same memory accesses every time. You can build analysis tools for CGC binaries; that’s what they're there for.

On the other hand, CGC binaries cannot manipulate, create, or even open files. Such is the price of idempotency.

CGC binaries can communicate over file descriptors, but depend on the CGC evaluation tooling to open them.

CGC binaries can be debugged with the version of gdb included in the DECREE VM.

Give it a shot!

We had two DECREE-based challenges in our 2015 qualifiers: cybergrandsandbox and patcher. As announced during the 2015 DEF CON closing ceremonies, the winning computer from CGC will be competing in our 2016 finals game. Knowing how to work with DECREE will probably be a valuable skill for quals and finals this year.

Quick Quals Qupdate

The DEF CON CTF 2016 qualifying contests are going very well. Here's where we are:

Contest Dates Notes
DEF CON CTF 2015 May 16 - Aug. 9 2015 Qualified DEFKOR
HITCON CTF Oct. 17 - Dec. 6 2015 Qualified blue-lotus
RuCTFE Nov. 21, 2015 Qualified StratumAuhuur
SECCON CTF Dec. 5 2015 - Jan. 31 2016 First place team already qualified
32C3 CTF Dec. 27 - 29 2015 Qualified pasten
Boston Key Party Mar. 4 - 6 2016 Qualified HITCON
PlaidCTF Apr. 15 - 17, 2016 Qualified LC↯BC
0CTF Mar. 12 - Apr. 24, 2016 First place team already qualified
DEF CON CTF 2016 Qualifiers May 21 - 23, 2016 Online Jeopardy style, qualifying eight teams
DARPA Cyber Grand Challenge Aug. 4, 2016 All-machine competition, on-site in Las Vegas

About SECCON CTF and 0CTF

We made the difficult decision to not accept any qualifying teams from these contests. While both contests went spectacularly well, the first-place teams from both contests were already qualified for DEF CON CTF Finals. Since both games had very few teams in their finals slate, and scores dropped off so sharply after first place, we've decided to not accept second or third place teams from these events.

Legitimate Business Syndicate has nothing but the complete and utmost respect for the SECCON CTF and 0CTF organizers: we know exactly how hard it is to run CTF games, and score them fairly. We hope to promote and work with them again in the future.

Quals are Coming

DEF CON CTF Qualifiers start on May 21, at the stroke of Midnight UTC, and run for 48 hours. We are planning on having a "Baby's First" category of approachable problems for hackers of all skill levels, and the usual slate of harder challenges that'll make you sweat, cry, and binge-drink your hacking beverages of choice.

Register today, or any time during the game.

Cyber Grand Challenge

We're excited to see what happens with the DARPA Cyber Grand Challenge Final Event at DEF CON. If you're interested in learning more about it, check out our CGC For Hackers series, that we will be adding to on the road to Vegas.

What is the Cyber Grand Challenge?

DARPA’s Cyber Grand Challenge is “a competition that seeks to create automatic defensive systems capable of reasoning about flaws, formulating patches and deploying them on a network in real time.” More laconically, it’s Capture The Flag for autonomous computers.

Just like DEF CON Capture The Flag (CTF), Cyber Grand Challenge (CGC) is a contest with two separate events. The CGC Qualifying Event (CQE) was held on June 3, 2015, and the CGC Finals Event (CFE) will be held on August 4, 2016, at DEF CON. Unlike DEF CON CTF, the competitors are Cyber Reasoning Systems (CRSes) that compete autonomously.

CQE had 28 teams participate, finding flaws in more than 99 of the 131 binaries during the twenty-four hour event. In addition to finding flaws, competitors patched binaries to remove vulnerabilities. Patches were graded on time, memory usage, and space efficiency. Competitors only had access to organizer-provided binaries, making this similar to a Jeopardy-style CTF, where competitors only have access to organizer-provided challenges.

Seven teams from CQE will be competing in the CFE in August. The scope and complexity is much bigger. Similar to an attack-defense CTF, organizer-provided binaries are just the beginning. CRSes will be expected to process binaries submitted by other competitors, in a complex “consensus evaluation” process that we’ll detail in another post.

The Cyber Grand Challenge is serious business. Not only does the top team win two million dollars, but every competing team will have pushed the limits of automated binary analysis and patching. In addition, the consensus evaluation format is a massive new development for the CTF community as a whole, and the popularization of CRSes for CTF competition will drastically change how CTF games are played.

Official CGC Links

Unofficial CGC Links

What is Capture The Flag?

Capture the Flag is a family of competitive games involving opposing teams trying to steal something of value from each other. In computer security, the flag is typically a piece of secret data, and the territory from which it is captured is a computer system controlled by the opponent. There are two popular formats for computer security Capture the Flag (CTF) games: Jeopardy-style and attack/defense.

Jeopardy-style CTF games have the organizers running a set of challenges that each team has to solve for points. This format is popular because of the relatively simple logistics, especially for online games. Challenges are generally independent from each other and ideally idempotent between connecting players, which leads to reliability and stability for large contests. Scoring is nice and simple too: add up the points for solved challenges, and use timing of solutions to break ties. The DEF CON Qualifier has traditionally been a Jeopardy-style game: the engineering and operations work that goes into a game for over a thousand teams isn’t easy by any means, but it’s not difficult either.

Attack-defense CTF games are more similar to CTF in other contexts: the opposing teams themselves control their flags’ defenses, and the organizers merely create and referee a fair playing field. Attack-defense CTFs generally have each team defending one or more vulnerable services, with the ability to attack other teams’ identical services over a network. This game typically requires more work to engineer and operate because the way teams interact with the underlying game systems and each other is much more complex and open to finding weaknesses with the rules of the game itself.

For instance, by destroying a system’s availability to process benign requests (i.e. taking it offline) you can completely rule out processing malicious requests that would compromise confidentiality or integrity. Organizers have to build, run, and score availability checks to require defenders to run potentially-attackable services, otherwise attackers won’t ever have the opportunity to attack a vulnerable service.

Modern computing environments have much more powerful defenses available: defenders can and have used virtualization to protect flags in memory, system call blacklisting to protect flags on disk, and other so-called “superman defenses.” Ruling these out to force defenders into a narrowly-defined set of “acceptable defenses” takes considerable organizer effort.

DEF CON Finals is traditionally an attack-defense game. The antagonistic nature of the game becomes a complex test of time management, creativity, and the skills involved with computer hacking that isn’t available in any other legal venue.

There are other kinds of CTF that can be run: some of us have played a game that could best be described as "king of the hill," and the DARPA Cyber Grand Challenge uses a format called "consensus evaluation." The latter will be discussed more here soon, as there are implications for the future of DEF CON CTF there too.