Introduction

First, I’d like to give thanks to the people that made this season happen. Massive thank you to Coach Carpenter for his guidance and support throughout the season. He’s the one who got us home from the airport after a very, very long and annoying series of flight gate changes, delays, and cancellations (a whole ordeal that’s a story for another time). I’d also like to thank the rest of the team for their hard work and dedication. A big thank you to all of the WRCCDC and NCCDC teams and staff for putting on these awesome competitions year after year. Finally, I’d like to thank the sponsors for making this competition possible.

I’d also like to shoutout sourque for his CCDC writeup and support. I used his writeup as a loose template for this post, and I’d highly recommend you check it out. We wouldn’t have gotten as far as we did without his help.

Lastly, this post is largely written as a narrative rather than a technical flyover like other posts may be. Sorry if it’s excessively long. I wanted to write this post to share my experience and perspective throughout the years, so I hope you enjoy it.

CCDC is a smelly cheese competition where the pompous "red team" stroke their ego on freshies and teach them to delete iptables!!!!

You don't have to cheese to win in CCDC. It just often tends to be easier to do so. I encouraged our team to do things as legit as possible and avoid doing anything too cheesy, and lo-and-behold we got 2nd nationally. At the end of the day, yes, CCDC is a competition, but if you take it with the perspective of learning real best practices, you'll get so much more out of it. I don't think there's too much the competition can do to prevent people from just winning via cheese or cheating, but I hope future teams will take the same approach. Besides, winning is only a small part of the learning experience that is CCDC.

If you aren’t already familiar with CCDC, here’s a quick summary from their website:

The mission of the Collegiate Cyber Defense Competition (CCDC) system is to provide institutions with an information assurance or computer security curriculum a controlled, competitive environment to assess their student’s depth of understanding and operational competency in managing the challenges inherent in protecting a corporate network infrastructure and business information systems. (via nccdc.org)

Breaking the Streak

Before we talk about this season, I need to explain a little bit about the history of Cal Poly Pomona’s CCDC team.

But what if I don't care about the history

That's ok. It's a personal story and shows why the past 3 years have been important to me. If you want to skip to this section, go ahead. The table of contents is on the right.

Precedent

CPP has been competing in CCDC since the second season in 2009. In the team’s second season ever, the 2010 CPP CCDC team placed 3rd in the national finals. While it wasn’t perfect, things were looking on the up and up. Unfortunately, that was the last time CPP would place in the top 3. During this drought of performance, CPP would not place in the top 3 for the next 13 years, and the CPP CCDC team hasn’t even been to nationals since 2016. The team would consistently be fighting at the top of the western region, but they would always fall short.

2020-2021 Season

My CCDC story began during the 2020-2021 season. The world was still grappling with the COVID-19 pandemic, and the WRCCDC was no exception. WRCCDC was held virtually, but I was still excited to compete. Throughout the summer, I participated in the bootcamp that the team held to pick the newest members, of which Alex and I became of after successfully completing the tryout. For the next 5 months, we practiced online in study groups focused around each main role of the competition: Linux people (me), Windows people, and Everyone Else™. As the youngest member of the team, I focused on trying to soak in as much as possible, like VIM and Tmux from Brice. I would do whatever was assigned to me and try not to contradict anyone because this was one of the best teams in the region, right?

While we tried our best, our season was cut short when we failed to make it past the western regional qualifier. This was a feat no CPP team had ever heard of.

2020-2021 WRCCDC Qualifier Results

2021-2022 Season

After the season ended, the next captain, Taylor, was selected and then we said our goodbyes to those who were graduating. Heading into the 2021-2022 season, we were just 3 members strong. Usually rebuilds don’t start performing well, but we were determined to fix what we had broken last season. We recreated the summer bootcamp from scratch, and despite still being fully virtual, we managed to recruit a full team of young, promising new members. In the fall, CPP decided to open the campus back up for school, so our team was able to finally have in-person practices. The difference was night and day. We were able to get so much more done in person, and we were able to get to know each other better. We did well in our practice competitions leading up to the qualifier, but we were still such a fresh team. When Alex decided to step down from the main roster, it was only Taylor and I left who knew anything about the real competition; even then, only Taylor knew about what lied ahead. At qualifiers, it was another year where we fumbled communicating (cough cough, runonce, cough cough databases), but we managed to catch the resulting mistakes late in the competition and resolve them. The main issue our team faced was that we were missing how the red team was continuing to get access to our systems. We ran through all our checklists, but it wasn’t enough. We only managed to salvage our systems because we understood how our operating systems worked. Despite the setbacks, we managed to crawl our way into the regional finals.

2021-2022 WRCCDC Qualifier Results

Just over a month later, we were on our way to the regional competition. We had a few more practices, but all of us were juggling a ton of other responsibilities which made it hard to really make groundbreaking improvement. Many late nights, or early mornings depending on how you look at it, were spent developing a new plan: dockerize and migrate all services. The theory behind it is that even if the red team got RCE through our scored services, they would end up in a useless container. We got the process down to a science in our lab environment and we were confident going into the regional finals.

Right out of the gate, we were humbled. Our plan to dockerize every service failed miserably when we just couldn’t figure out how to get the networking to work right. Between endless domain replication issues and having to reinstall operating systems multiple times because we forgot to install the bootloader, we couldn’t catch a break. Due to all the chaos, we didn’t even realize we forgot to change the password of the router. Remember how Taylor was the only one who had been to a regionals before? Well, he was out doing inject presentations for several hours. Without any real form of leadership structure left or prior experience, we were dead last in services after the first day. We went back to the hotel and managed to figure out the competition environment, but we were all exhausted. The second day, we swung back with signs of life and climbed back all the way to 3rd in services by around noon. Our hopes were high, but the default password on our router was exploited on the second day and we paid dearly for it. Our scoreboard was a sea of red and there wasn’t anything we could do about it. No matter how hard we tried, we couldn’t get back into the competition after taking those massive losses. Heads hung low, we reverted what we had to and went through the motions we had practiced, but we knew it was futile anyways. At the end of the second day, yet again, we were dead last in services and the gap was even larger. To no one’s surprise, we were last place overall. While Alex, Taylor, and I were marginally happy that we did better than last year, we were still disappointed. We had a great team, but we just couldn’t get our act together. This would be both the last season for them and four other team members, making me the sole remaining member of my original team.

2021-2022 WRCCDC Regional Results

Slash and Burn // PRESEASON

Before we left the awards ceremony, Taylor announced he would pass the torch down to me, and I became the new captain of the CPP CCDC team. I was determined to learn from all the mistakes of the past and not let the team fall apart like it did year after year.

It was time to rebuild the team from the ground up yet again. I wanted to make sure that the new team was ready for anything that could be thrown at them. I wanted to make sure that the team was ready to win. The incumbent members got to work quick; our broken pride had lit a fire underneath us. By the very next week, the bootcamps for the summer entered their planning phase. Going into the next season, we completely cut our old ideas of how CCDC worked.

What did you do in the past?

The approach of the first CCDC team I joined was to secure every box from the ground up. This means secure every service, config file, permission, and user on every system, because if there are no vulnerabilities, well... then there are no vulnerabilities. This means that you have to research all known vulnerabilities of all possible services for CCDC. While this approach can be very useful to learn how systems/services work, the problem with this approach is that it does not scale well unless you script out everything. Yet, in years past, the team leadership refused to publish scripts out of fear of giving other team's the game winning strategy/tool. In reality though, who could possibly use a tool/strategy better than the team that made it? And so, we decided to oust this short-sighted axiom.

We completely revamped the roster composition:

1. Team Manager (Linux/Business)
2. Business Lead
3. Business (Networking)
4. Networking Lead (Windows)
5. Windows Lead
6. Linux Lead
7. Linux (Web/DB focus)
8. Linux (Linux support/threat hunting for both Linux and Windows)

To scout for new members for this composition, we rebuilt the summer bootcamp from scratch again. More labs, more homework, more time spent on making the bootcamp content flow coherently. The whole process was made fully public and uploaded onto our website. Feel free to use this as inspiration for your own bootcamps. We wanted to make sure that the new members were ready to go out the gate as long as they put in the work. We had a great turnout (100+ participants), and after 8 long weeks of bootcamping, the top crop of the tryouts were selected.

Since the cybersecurity clubs at CPP tend to share the same people (there’s only so many places passionate people can be), many of the new members of the team had already interacted before. Not having to build up team relations from 0 was definitely a boon, but it’s not like we were all “friends” just yet. It’s pretty hard to overstate how important it is for a team to mesh well together. To help everyone get to know each other, we split up the team of 12 into 3 teams of 4 to compete against each other in Hivestorm, a tradition that I picked up from my first captain.

Hivestorm is literally a CyberPatriot round. Nothing too special, just a basic find-and-fix competition, but it helps get new members into the competitor mindset and begin to perform under pressure. It’s only 4 fours, but it helped us get a feel for how everyone works together and using checklists. A quick W, but a journey of a thousand miles begins with a single step.

As captain of Team 270 (YOU JUST GOT TANAY'D) I just want to say a big thank you to my teammates from @calpolyswift! Our team had lots of fun. Let's see who gets TANAY'D next year 😉

— Altoid0 (@Altoid0day) October 16, 2022

One of CPP Cyber’s CCDC sub-teams, “YOU JUST GOT TANAY’D”, winning Hivestorm

With the warmup behind us and 1st place secured, the WRCCDC invitational round was quickly approaching.

Remember how I mentioned that most of my team are involved with campus cybersecurity clubs? Well, many of us are specifically heavily involved with the cybersecurity club, SWIFT, at CPP. We host weekly workshops, CTFs, and other events. One of the semesterly events we host is Red vs. Blue (RvB), which is more or less a CCDC-inspired event. How does this affect the team? Well, due to issues with our infrastructure (thanks rain), we had to reschedule SWIFT’s Fall RvB to the same day as the WRCCDC invitational. This meant that we had to split our team in half. Some of us would be competing in the invitational, and the others would be running RvB. This was a huge challenge for us, but we were determined to make both events work. We had our team members swap in and out throughout the day, and by the end of it, everyone had a shot to run through their game plans, see what worked and what didn’t, and experience what CCDC is actually like. While we were able to get 2nd in the invitationals, almost all teams were experimenting with different strategies, technologies, and rosters, so the results couldn’t be taken at face value. Still, the fruits of our labor were beginning to sprout.

2022-2023 WRCCDC Invitational Results

The Fresh Blood // QUALIFIERS

After a eventful winter break, the WRCCDC competition season was finally here.

What happened during Winter Break?

I also compete in CPTC, and the global finals took place in early January. We became back-to-back champions! You can read my teammate's writeup about that here.

In the weeks leading up to the February qualifier, the OS teams focused on understanding the services that were teased by the Operations Team like Kubernetes, web applications, and mail. The networking team was developing strategies for detecting malicious traffic and blocking IPs. The business team and I were busy using previous years’ injects as inspiration for research and templates so that we could minimize the amount of novel work we had to do during the competition. I tasked one of my teammates, this year’s Linux lead and my former Linux subordinate, Dylan to learn Kubernetes in the short time we had leading up to the competition. While he hated it for all its quirks and frustrations, he was able to get a basic understanding of it and was able to use it. Once the first month of our semester finally ended, it was time to compete.

We were confident for sure. Not everything was perfect, but we practiced what we wanted to and had the faith in each other that we would showcase our potential.

The morning of the competition finally came, and we all arrived nice and early in a meeting room of the IT department at school that we were permitted to use. The U-shape of the table space made it optimal for team organization. For our team of 8, the Linux team was on the left, the Windows and Networking was on the right, and the business team and I were in the middle. We sat at our table on Discord awaiting for the in-brief as we were all excited to get started. All our laptops were set up and we had the inject portal loaded up on our screens. The Operations Team began the very short in-brief which was really just a way to make sure everyone was ready to go and officiate the beginning of the event. The competition was only delayed a short while: our real start time ended up being 9:05AM. Once the competition began, we attempted to access our router to change the password and start applying our firewall rules, but right out of the gate we had problems. We couldn’t access the router. Well, we could ping it at first, but quickly we weren’t even able to do that. Then, we saw the scoreboard as the first 5 teams (teams 1 through 5) had all red across the board. The red team had taken down all the routers immediately via an untested script they made the night before. Due to this unfair event, our routers were reset, and we were given grace periods to recalibrate. The red team had a firm talking to about ensuring their scripts are well tested and don’t inadvertently DOS the entire environment. And so finally, we were able to actually get started.

The OS teams were able to focus on their systems and services, but the networking team was struggling to set up the router. For some reason, even after the reset, we were not able to access the router shortly after it was reverted. For the first 3 or so hours, we had 0 router defense. Eventually I requested a consultation as to why our router kept breaking a couple of minutes after restarting. After some kerfuffling where Karina was configuring our firewall too fast, because the operations team was getting blocked by our ingress rules (she was so fast they thought she was a script 💀), we discovered the root cause. The reason? We were whitelisting the wrong IPs for router access because the IPs we received in our email about competition environment information had a typo in it. So, even though we were typing it in as outlined to us, we were accidentally blocking ourselves from the router. Having that cleared out of the way, we were finally able to start applying our firewall rules without having to revert it constantly.

The whole while, our services were mostly stable. On the Windows side, FTP would occasionally go down, but we were able to restore it quickly. On the Linux side, we had a couple of services go down, but we were able to restore them. Kubernetes was a Rancher deployment with several containers running various services, all connected via KeyCloak. Dylan’s research paid off as he was able to navigate and figure out how the whole deployment worked after a few hours, and even discovered secret passwords that weren’t previously given. Unfortunately, the Operations Team realized their mistake and made the secret known to all teams via a public announcement, so our hard-earned advantage was lost. Kubernetes definitely showed to be a tough beast to manage as the teams struggled to understand it. Eventually, Dylan and our Windows lead, Tanay, identified that the deployment was also connected to the main AD server for DNS, so we were able to resolve our dependency issues and secure that connection. As we were nearing the last hour of competition, our subnet blocks had effectively made red team presence nonexistent. The only way they could affect our scored services was via logging in with default credentials and trying to mess with our web servers. Across the board, we did struggle with properly communicating between team members which services had their passwords changed, and which passwords were in current use. Fortunately, I made a quick script to automatically re-deploy database backups, so their red team efforts were (again) contained without us needing to worry about password changes.

The only thing we had to worry about was injects. We were able to keep up for the most part, but the sheer number of compliance-based injects made research a serious time crunch. We had to rate ourselves against several compliance frameworks and, to be honest, it was laughable at best. We were given way too much, with way too little time. Not only that, but there’s no way we could know all the compliance frameworks/have templates ready to go (which is advice that WRCCDC literally gives every year) unless we had come to expect this from previous years in addition to having established experience pipelines. This makes the competition extremely difficult for new teams for no good reason, in my opinion. But we did our best and submitted what we could until the very end. Qualifiers were exhausting, with plenty of our patented User Issues™ issues, but we felt like we had done enough to make it into the top 3 for certain, with a competitive chance of getting 2nd. It would have to end up being a close call on inject performance, but we figured other teams couldn’t have done too much better except for Stanford who were far and away the 1st place team.

This year the results were revealed live over an awards show/draft show of sorts organized by the WRCCDC Operations Team and was streamed on Twitch and YouTube. Because of this, we had no idea where we ended up in the grand scheme of things. When the broadcast started the next day, my team and I were all eagerly hoping to hear our name. First place went to Stanford. We knew that was inevitable. But then the real shocker hit. We got 4th place overall.

2022-2023 WRCCDC Qualifier Awards Show

The difference between injects and services made no sense to us. It was clear that injects influenced the final placements the most. Even though we were 1st in defense, it made no meaningful impact. We had substantially better services and defense than the 2nd and 3rd overall teams, but injects were just a tad off and yet they placed above us. There was just something we couldn’t put our finger on. Injects used to be our strongest point, how could we have fallen so far? We were all disappointed, but we knew that we had to keep our heads up and keep pushing forward to make sure that we were ready for regionals.

2022-2023 WRCCDC Qualifier Results

The Gauntlet // REGIONALS

Arrival

When we arrived at the hotel, we were greeted by The1ProBro and received our competition binder. Included was basic information about the environment like lore about the fictional business, credentials to the inject portal, and a very, very low res topology with non-standard symbols for systems.

After getting settled in, we consolidated in my room to go over the binder and discuss our game plan. For the majority of the rest of the day, we analyzed everything in the binder and tried extrapolating everything we could knowing that there was information purposefully left out. While analyzing the topology, we realized that the environment would be a combination of this year’s invitationals and qualifiers environments with a few added elements. We were given 1 inject to complete that night which was to prepare a presentation detailing what our team’s plan was for the C-suite of the company. We designed a presentation but didn’t submit anything since the inject didn’t actually ask for anything to be submitted. We also began to write down some specific game plan elements with the information we had like injects we would expect, firewall rules we expected we would have to add, or other things we would need to information gather in everyone’s personal binders/notebooks. Midnight came sooner than I expected, so we decided to call it a night after going through everything one more time.

Day 1

The first morning of a competition always feels different. A blend of excitement, anticipation, and a dash of nerves. It’s something I never get tired of. My team met in the lobby to eat the hotel breakfast, but the line was way too packed with all the teams and other hotel guests. So, we gathered our things and headed directly to the competition which was held at Coastline Community College again this year. On our way there, we stopped at a nearby McDonald’s for breakfast. I got a sausage McMuffin and a hashbrown. We arrived at the competition, checked in, and waited in the lecture hall until the in-brief began. While getting our badges during check-in, we were informed that due to some last minute “supply-chain issues”, WRCCDC had to switch badge companies last minute which resulted in the Orange Team and Red Team badges having a near identical color (and no team name on the badge, which will be important later). Other than that, check-in went smoothly.

During the in-brief we were informed of several game-altering announcements:

1. Google Drive would not be received until further notice (normally it would be received at the beginning of the day)
2. USBs would not be received until further notice (normally it would be received at the beginning of the day)
3. We would not have access to sign-up for anything due to not having the Google Drive email account
4. There was a lack of volunteers present at the beginning of the day, so coaches and alternates were asked to be room judges at first
5. The presentation inject would require presentations basically immediately after the competition started

To put it lightly, we were not expecting any of this. A good portion of our strategy relied on these things not happening. I know first-hand how it is to run these kinds of events and sometimes things happen, so we won’t complain too much. We just had to adapt.

We rapidly had to start adjusting our game plan to avoid using Google Drive and even USBs.

Room Judge Shoutout

Big shoutout to the coach of Saddleback College for being a great room judge and supporting us like their own. Later on, we got a different room judge who was also great and pretty funny. Legendary read on Dylan to instantly guess his favorite food is sushi. Really awesome people. 😊

We went up the stairs to the classroom for our team and got settled in. A lot of things were about to be on the fly.

The physical environment was set up as 3 rows of computers. The front row had an IP phone, a Cisco switch and router, an IP web cam, and 3 desktop computers (some ESXi). The second row had 3 laptops, a wireless router, and a desktop. The third row had 2 more laptops. Our environment wasn’t anything too unexpected, but we weren’t able to access the desktop that was hosting the web interface for our web camera. Considering the lack of lateral movement in the environment as far as we could tell, we didn’t think it would be too much of an issue and left it alone for the entire competition. The web camera was a bit of a concern, but we decided to communicate primarily over physical means with printed papers and sticky notes. This allowed us to avoid leaking our passwords by writing it on the whiteboard or by speaking it aloud. Besides, our main goal was to rotate passwords of all services, web apps, and users anyways. The Windows side of the environment was very predictable and was pretty much entirely scripted out. There wasn’t anything substantially added to the environment that wasn’t from the qualifiers and invitationals environments. It was only the Windows servers operating on the ESXi in our first row. The Linux side had a few unique systems and services, but they didn’t affect the scored services so I reinstalled several systems with a standard Windows 10 OS. The less weird, annoying quirks set up by the ops team the better. We were able to identify that the remote location (qualifiers environment) wasn’t really dependent on things running on our on-prem environment, so we were able to safely stick to our game plan for the most part.

There were a few scored services in the environment we couldn’t locate in any physical system nor virtual system. We knew they were there based on the scoring engine and presumed they were in AWS, but due to a permissions issue with our provisioned AWS account, we couldn’t view them in the AWS console. We had to bite the bullet and wait until the operations team got it sorted out. Unfortunately, we were also unable to log into our router. We tried every password we could think of, but nothing worked. We were able to get into the switch, but we couldn’t find any way to get into the router. We decided to leave it alone for the time being and focus on other things since the default credentials didn’t work for us so they shouldn’t work for the red team.

Other than that, regionals was fairly standard for WRCCDC. Our business lead, Jessica, was called for presentations twice (being gone for upwards of 2 hours each time and unable to contribute to our injects) meant that I had to shift my priorities to injects and to answering phone calls for the majority of the early competition. We learned our lesson from last year, and this time we were ready to shift such roles fluidly.

One of the services that went down was a Linux web server that wouldn’t restore even after a reset. I suspected potentially a dependency was red teamed (like a database), but since I was focused on other things I couldn’t personally investigate. I tasked the Linux team, who were coordinating all our Linux systems, changing web app passwords, and managing Kubernetes services, to shift priorities to the one service as everything else seemed to be in order. Unfortunately, we were unable to resolve the issue by the end of day. For all other services that went down, we were quickly able to identify the reason and patch the issue with a firewall rule or a service restart.

When we left the room at the end of the competition time, we were feeling pretty good. We had a few services go down, but we were able to recover from them and we were able to get a lot of injects done. We went back to the hotel, got some dinner, had a quick debrief of our unresolved issues and goals for day 2, and went to bed early to get some rest for the next day.

Day 2

The second day of competition started off fine. We woke up well-rested, took our time getting ready, and waltzed into the lobby wondering why no other teams were prepping to leave. On our way to Coastline, we stopped at McDonald’s again (more sausage McMuffins yum) and ate outside the building.

But it wouldn’t be a WRCCDC competition without a “WHERE’S CALPOLY?” moment.

See, the night before I told my team that the in-brief for the second day would start at 8:30AM because I was referencing a hypothetical schedule posted in Discord before we got a real schedule. At that point I was so tired that I forgot we actually did get an official schedule in the competition binder we got when we arrived with official times for when events started. While eating outside the entrance, our room judge from the previous day came out of the building to grab something from his car and told us he was proud of us being second in the standings. We were confused. How did he know that?

“Oh, it was on the scoreboard in the in-brief. Everyone was just in the room.”

Even more confused we asked if teams were allowed to go in before the brief started, since it was totally supposed to start at 8:30, but he confirmed our worst fears: the in-brief started at 8:00, and ended at 8:30. Meaning the red team had just started, and we weren’t even in the building yet.

The schedule I followed even though I had the real schedule…

And that’s how we started the day. We rushed into the building, Dr. Brown yelled “THERE YOU ARE CALPOLY”, and we ran to our room. We were about 10 minutes behind schedule, but we were determined to make up for lost time. We had a lot of ground to cover, and we had to do it fast.

Just as we were getting reaccustomed, a random guy walked into the room and demanded immediate access to our WiFi network. Because we missed the in-brief we had no idea if social engineering was now in play, so we were skeptical if he was Red Team or Orange Team. The badge color issue I mentioned earlier didn’t help. I approached the situation entirely in-character at first, but he quickly broke out of character when he realized we genuinely weren’t sure if it was authorized or not. We ended up having to make several phone calls with the White Team and Orange Team to confirm that he was indeed an orange team member and was not social engineering us. While he was kind enough to not penalize us for being excessively cautious (it’s a security competition after all), it still distracted us from the competition and wasted a lot of time. All the while, the Operations Team finally got around to completing our revert ticket, but they reverted our DC instead of our router.

How could they have made such an irresponsible mistake?

First of all, the Operations Team was very swamped throughout the competition, so obviously we were aware such a thing could happen. As with any other interaction, we handled it professionally, and patiently informed the Operations Team of the situation and it got resolved. They said they would award points back, and give a brief grace period from the red team.

So, what actually happened? Stanford had submitted a revert ticket for their DC at the same time we submitted ours, and they just had a mixup with the team numbers. Funnily enough, we made the same exact mistake when our team was hosting our Fall RvB event during the WRCCDC Invitationals, and accidentally reverted the wrong participant's box. So we definitely understood that sometimes it just happens.

We decided we would just leave the router as it was since it wasn’t causing any problems, and at this point trying to fix it was proving to be a lot more trouble than it was worth. We had a lot of injects to do, and we were already behind schedule so taking out the internet wasn’t a favorable plan.

After a short while, Dylan was able to identify that I was correct in my assumption that the Linux web server was missing a Postgresql database that was pwned. He was able to get it back up and running, and we were able to get the service back online.

After that, we were pretty much on a pretty set course of action. We were grinding through injects and our services hardly went down after we finalized and double-checked our local firewall rules across everything. Angry customers were tamed, confused employees were made clairvoyant, and tickets were being filled.

Although the AWS permissions issue was fixed late the previous day, we didn’t really dive into AWS until the 2nd day. We found a deployment of containers that matched the scored services we were missing. They were really shoddy Python Flask apps (like a calculator) that were clearly vulnerable to simple command injection. While we couldn’t figure out how to use the CI/CD pipeline the operations team had set up to deploy the containers, we figured that we could simply just use a real-life strategy of load-balancing via multiple instances of the containers. We had like 5 instances of each container running, and whenever a service went down, the next check would likely hit an untouched instance of the service and we would just redeploy the affected containers. It wasn’t great, but it would indefinitely prevent SLAs. Happenstance or not, the red team didn’t attack the AWS scored services until late morning of day 2, so our late start for AWS didn’t really hurt us.

With these adaptations, we were on track to take over Stanford, the team in 1st place services. But, as luck would have it, our router finally gave out to whatever was wrong with it. We still couldn’t figure out what specifically had gone wrong, and we didn’t have internet access to make a revert request. It was so late into competition, anyways, so we didn’t feel like it was worth it to try and fix it. Everything on the co-location was still fully operational, so we just decided to focus on injects until the end of the competition. My business lead and I resorted to physically running back and forth between the White Team room and our room to submit the final handful of injects. We were able to get a lot of injects “done”, but they lacked quality. Due to time constraints, we had to rush through them and put the documents on the USB and pass it around to copy-paste the contents into a master doc before converting to PDF for submission. Once we finished the final inject, we had nothing to do but wait. Leaned back in our chairs, we stared at the clock.

5 minutes left. 4 minutes left. 3 minutes left. 2 minutes left. 1 minute left.

Hands up. That’s all we could do.

Service status at the end of the competition

Uptime percentages after day 2

Awards

This year, the WRCCDC team were able to bring back trophies for category winners. The categories were: Best in Defense, Best in Services, Best in Customer Service, and Best in Business.

I was starting to have flashbacks to last year’s awards ceremony. But the growth was made immediately clear. We placed in every category. 1st in Support, 1st in Defense, 3rd in Business, and 3rd in Services.

We took every trophy with pride. We were so proud of our team. We had worked so hard. Though, every placement that wasn’t 1st felt like a knock on the chin. After all the celebration over the minor victories, we still had the most important placement to go. In that moment, we knew that we had done well, but we didn’t know how well, and more importantly if we had done well enough to overtake the big dogs of Stanford. First place is a direct bye to nationals, but second still gets a last-chance shot in the Wildcard Round, where all the CCDC regions have their regional runner-ups duke it out for the last seeding into nationals. Then, Dr. Brown announced the final standings.

“In 3rd place,” I held my breath. “…Arizona State University!”

My heart was beating out of my chest. In my head I remember thinking, “If we weren’t third, then we were top two. We had to be. We made it to nationals, then? I should be happy.”

“In 2nd place,” I closed my eyes. “…Cal Poly Pomona!”

It was weird. I was happy, but I wasn’t. Sure, I was happy that we made it to wildcard, but I was disappointed that we didn’t get first. We received our silver medals, held our big trophy, and took our team photo. It was so bittersweet to know we had done well but fell short of our goal. Personally, I felt that we would have done better in our inject scores if I had gone to do the presentations instead of delegating that to our business lead and I regretted that. If you looked at our expressions, you would have thought we performed just as well as last year.

But we weren’t out this time. We still had a chance to make it to nationals through the wildcard round. All we had to do was do what had never been done before: win the wildcard.

2022-2023 WRCCDC Regional Results

The Wet Season // WILDCARD

3 days. We had 3 days to prepare for the wildcard round. WRCCDC is usually the latest region to hold their regional competition, so we had the least amount of time to prepare for the wildcard. The Wildcard round is a entirely different style of competition. It’s exactly Hivestorm (and we won that). ‘Nuff said. The same style of vulnerabilities, the only difference is that there is an added CTF component to the competition. Most of our team had competed in CyberPatriot at the national level before as well, so we were all very familiar with the “find-and-fix” competition. If anything, it was perfectly our comfort zone.

For the team members who were still new to this style of competition, we put together a mini-bootcamp to get them up to speed. We ran through practice images, methodology, and even speedruns for fun. We didn’t sleep very much those 3 days, but we could definitely secure a server.

That Wednesday, we were ready to go. Back in the same room we had competed in for qualifiers. This time with Porto’s (the best pastries/baked goods).

For this year’s wildcard, there were 4 Windows images and 4 Linux images, so a nice clean distribution for our team. While we started off slow compared to other teams, we picked up the pace by the end of the 4-hour competition, getting large swings in points in both the CTF and image scores within the latter half of the event. Our Windows mains had a 95%~ average, and our Linux team had an 80%~ average. Personally, it was a lot of fun to jump back into what I had done for 4 years in high school and prove I still had it. Whipping out the notes from my CyberPatriot days showed the growth I had made since then, when I was a no-name, bottom-of-the-barrel Linux kid.

The CTF wasn’t anything too crazy, some forensics and some cryptography. I was working on the cryptography section when I needed a break from the Linux images. The crypto challenges were a set of 6 XOR’d files that needed to have each key brute-forced using cryptanalysis. I was able to get 5 of the 6 keys, but the last one was a bit of a doozy. You were given the SHA256 hash of the original file, so you could verify if you found the right key. The flag was just the MD5 hash of the file.

We ended with the 2nd best image score, and the 4th best CTF score, but the overall highest.

With this victory secured, CPP became the first WRCCDC team in CCDC history to qualify for nationals through the wildcard round. The 7-year long nationals appearance drought was over. It seems like the wet season was about to begin.

2022-2023 WRCCDC Wildcard Results

The Comeback // NATIONALS

Arrival

Our trip to nationals was pretty straightforward. When we arrived at Dallas, we were greeted by some NCCDC sponsors and volunteers at the hotel which was at the Dallas/Addison Marriott Quorum. After a day of traveling, we were tired and hungry for some Texan BBQ. Unfortunately, this is basically the only team picture we were able to take before competition, but I hold it near and dear to my heart.

The team having some mid BBQ

After we ate, we picked up some supplies like a whiteboard, pens, notebooks, etc. and headed back to the hotel for last minute preparations. We had a team meeting to go over our game plan for the next day and print out any last-minute documents. Originally, we had no access to prior years’ briefing packets the whole season, so we based all of our assumptions off of word-of-mouth comments from previous competitors. However, we discovered a repository that collected them. While combing over all the packets, we noticed that the competition has changed very little in recent years. The years of the pandemic left a lot of gaps of any potential recent changes, but sourque assured us it was still the same competition. Armed with that information, we drafted a plan that was much more tailored to a predicted environment than our more generic frameworks we had prepared for regionals. The environments tend to boil down to 3 networks that are all connected to each other. The first network is the “on-prem” network which contains the physical laptop workstations for competitors and usually also an ESXi with VMs. The second and third networks are usually fully remote ESXi servers filled with more VMs.

Day 1

The morning went by pretty quickly. We went to the hotel’s ballrooms which had been converted into the competition area. We were greeted by the NCCDC staff and volunteers who were all very friendly and helpful. We were given our team shirts and name tags, signed this year’s nationals banner, and had our pictures taken before the breakfast tables opened up.

While eating breakfast, we finally got the coveted briefing packet. We scrambled to read through it as fast as possible to get a feel for the environment. There were a few odd bits of information we weren’t expecting, like that every system need to have outbound connections to a specific IP, but nothing was too crazy. The CCS agent was something we were familiar with from the wildcard and our prior experience from CyberPatriot. Except this time, it literally only need to connect back and didn’t even score anything. It was just there to make sure we were doing things right. Another boon for us was the packet’s inclusion of scored service associated with an IP. We didn’t even have to discover it ourselves, which made things really easy.

Environment Diagram from the 2022-2023 NCCDC Briefing Packet

Lucky for us, the environment was very similar to what we had predicted using the packets of prior years. A big win for us.

Then, after all the initial speeches and sponsors, the real competition briefing began. Dwayne called all 10 team captains to the front of the room to pick our team numbers. With an excellent prediction maneuver (picking the far-right slip) we got Team 10. Usually team number shouldn’t matter, but as we saw at qualifiers, sometimes things just happen to hit low number teams first, so this was already a lowkey win for us.

Then, Dwayne explained a little about the competition and opened it up for questions before we officially began. The whole team was on edge, but we were excited to get started. Everything was falling exactly into place.

The first thing we did was secure our physical laptops with very simple firewall rules that basically boiled down to: deny all inbound. These physical workstation hosts had no scored services, so we were able to basically just firewall everything off and be safe at.

Generally, the first day of nationals sees very little red team presence. This year was not that. The red team came out of the gates swinging with immediate Eternal Blue and dropping payloads on our Windows infrastructure. Thanks to our Threat Hunter, Justin, we were able to detect it and respond quickly. Due to the nature of the early game, we focused on minimizing risk of persistence rather than IR reporting. Luckily the environment this year wasn’t configured to the point of unusable like we heard about in past years, so we were able to work quickly through our game plans.

The Linux side of things was very traditional in my experience. Lots of dependencies connected to a central database and a bunch of web servers (with a few other random things here and there). We noticed quickly that default credentials were gonna be a problem considering that there were several web apps running that all used the default credentials, but our WebApp lead, Jimmy, was busy running through our initial checklists with the rest of our Linux team. Due to the scale of the environment, by the time we got to securing the web apps, the red team had already made their presence known on the database box. We attempted several band-aid fixes, but ultimately didn’t do anything drastic like reverting the box to a snapshot. We were confident that we could recover from the damage and that the red team would be more focused on other things. Both Windows and Linux web servers depended on the database, so we were especially cautious whenever making database changes like changing default passwords, swapping database users to prevent credential reuse, etc.

Our threat hunter set up Wazuh to double as a central logger/SIEM and as a means of IR detection. By about midway through the competition, we were able to basically completely eliminate Windows vulnerabilities. Using our OSINT skills, we were able to identify how all the open-source services worked and were then able to secure them.

The business side of this competition was way better than in WRCCDC. Many more technical/interaction based injects made it a much better experience for the whole team (especially our business lead, who was in compliance jail for all of WRCCDC this year). There was only 1 presentation that lasted like 10 minutes from the time our business lead left and came back. If you want a complete list of injects, I’ve included them below:

Nationals Injects

Confirm that all systems were using the listed primary and secondary DNS servers. If not, configure it to do so and note the change.

Set up "Infirmary Integrated" to start receiving specific information about the patient's health on three different systems (one on each domain).

Ensure that all Windows machines are domain-joined. If there are boxes that are not, domain join them and make note of it.

Take into consideration the firewall equipment made available to us and propose a centralized logging solution that is compatible. - Configure it properly and show traffic from multiple firewalls. - List top 10 sources of network traffic across each of the three domains.

New member of the team requires SSH access. Inject provides us with their public key which needs to be configured on all SSH services.

Perform an audit of all user accounts. Identify and take note of all anomalies using the HR system as a point of reference. Do not remove any instances of anomalies yet (can disable the accounts in the process).

A member of the team got promoted and now requires administrative access to the three EMR web apps. Create a one page cybersecurity flyer to be posted in the break room.

Create a 7-8 minute presentation on our HIPAA compliance status. - Are we compliant? If not, what steps do we have to take to become compliant? - What is our plan of action if PII, PHI, or other sensitive information becomes compromised? - What are our legal obligations if we do get breached?

Create an RDP Jumpbox with a given IP address. Allow orange team access to Windows Servers from Jumpbox. Do not disable RDP unless otherwise told.

Create an incident response report on any red team activity we spotted on our systems. This incident response report is to be distributed to other teams for information sharing.

Determine the scope for a red team assessment.

Conduct a risk assessment of our network.

Repeat of Inject #2 for Day 2.

Vote on the top 3 incident response reports.

Go in for the presentation at your allotted time.

Remove all instances of unauthorized accounts and report them.

Configure SSH to block IP's for 5 minutes after 6 failed attempts in 60 seconds on all scored SSH services. - Report any instances of employees accessing social media platforms - Log and block access to specified social media platforms. Instagram, Twitter, Tiktok, etc.

Migrate a service from one box to another. OR migrate the service from one application to another. OR containerize the service. Your choice. Explain why you chose your migration method. The service must still maintain the same public IP after migration!!!

The user ""kogami"" expressed concerning statements during their off-boarding meeting. Brings up concerns of insider threat. - Remove all ""kogami"" accounts - Investigate logs for unusual activity for ""kogami""

Check for traffic from given IP address. Note any anomalous activity.

Inventory the MAC Addresses of all devices and explain your defenses against Layer 2 attacks.

Give your recommendations on outsourcing firewall management to a Managed Service Provider.

How would we assess each risk assessment proposal

Provide guidance on which firewall vendor to use for all network firewalls. Choose between Cisco and Palo Alto, explain why.

Gather a list of the top 10 ports and IPs seen in network traffic.

I mostly focused on getting the team through the early phase of the competition where it feels like things are coming from a hundred angles. I wrote notes about the environment, drew a topology, and marked any activity we noticed. My manager role was designed to give an individual the space to focus on the big picture and to make sure we were all on the same page. For example, communication directly between two people isn’t always clear or effective, but a third-party listening in can identify the issues and open up the information silos that tend to be built up throughout a competition. The manager being the central flow of all information also makes it a pristine option for answering the IP phone (which was graded customer interactions). The manager will be more likely to answer questions or be familiar with who has the required information as opposed to a random team member who focuses within their bubble.

The LED service boards in Dwayne’s headquarters are a fun and IMHO cool way to represent the competition-wide scoreboard for teams. Unfortunately, they served to be more of a distraction to teams. It’s hard to focus on the competition when you’re constantly leaving your room to go look at it. I made it a point to minimize our time spent looking at it, but even I couldn’t help but go out to see how our services were doing relative to everyone else. Everyone seemed to be hit evenly, though, so it was definitely a waste of time to go out and re-verify things we already knew. Don’t get me wrong though, it’s a cool idea and I’m glad they do it, but teams need to not fall for the bait.

At the end of the first competition day, we still couldn’t stabilize the database box’s defenses (it was frequently being targeted for obvious reasons) which left a lasting impression in our service score for the first day (as its tendrils happened to affect multiple services). The rest of the Linux environment was pretty much locked down to the best of our knowledge. The entire Windows side was basically untouched, and we were confident in its defenses.

We had a working dinner where we discussed all the observations we made about the environment. While in the heat of the competition, it’s so easy to get tunnel vision and miss the bigger picture. We realized that the entire environment was full of self-contained servers which means they only needed to allow traffic with the competition servers. We redrafted our local firewalls and basically focused all our defense into the database server. We printed new documents so that everyone had a copy of the topology and a mapping of services to IPs, then ended the night with a clear path forward.

Day 2

We were all sleepy on the second day. Some members couldn’t help but fall asleep during breakfast. But since we were decently confident from the first day, we allowed ourselves to be more relaxed.

The second day’s brief was much shorter. Very quickly, Dwayne revealed the scores from the previous day. Somehow, despite all our efforts, we didn’t place well in any category. This woke us all up. We had to really kick into high gear to gain some ground.

Right out of the gate, I started to resecure the database server, but the red team was much more aggressive than we expected. Perhaps it was because they didn’t have a ton of activity elsewhere, but they continued to hammer the database over and over again. Our SQL backup wasn’t poisoned, but we didn’t have a copy of our latest version that wasn’t already pwn’d. So, we had to use an old version of the database, but that led to a bunch of credential reassignments because it was before we employed per-service DB users. So, we spent at least an hour going back and forth between credentials and backups. At a certain point it proved to be too much of a hassle.

While all of this was happening, the rest of the Linux team was working on mass deploying scripts which led to our Linux main opening up their laptop’s firewall so that our web admin could get some files. However, we forgot to turn the firewall back off, so the red team was able to get a foothold on the laptop. We were able to quickly identify that they deployed Sliver, so we were able to quickly remove it, resecure the laptop, and submit an IR report. The red team didn’t seem to do much with their beacon, so it was easily mitigated.

About mid-way through the day, while checking on the LED boards, we got unexpected, good news. The active incident light was off (meaning it was successfully mitigated). Apparently, this incident was a rogue admin that we had to remove from our network. We were the only team to discover due to documenting our actions, which meant that we were able to get it back after reverting. The only other team that managed to get the incident did it by accident, so they weren’t able to get it back after they reverted. Documentation and team communication are important. The active incident was worth several hundred points, so we knew it was a significant swing to hopefully narrow the service gap.

Our threat hunter wasn’t too busy considering we had very minimal incidents that he could actually affect. The database would seem like an obvious choice, but we couldn’t deploy a Wazuh agent on it due to it being incompatible for some reason. So he was mostly just monitoring the network and the Windows side of things. Fortunately, LSASS calling out to localhost kept him busy for a while. This false positive was just enough to distract us from real red team activity where he left a Python HTTP server running, which had credentials for the Wazuh deployment in an install script and let them deploy an ARP storm across our network which seriously messed with our internal network. The Linux team had to rapidly flush our ARP tables to fix the sinkhole and shut down Wazuh entirely since there wasn’t much use if there weren’t reportable incidents.

Our network lead, Evan, was focused on dealing with a defective Palo Alto firewall that was causing a lot of issues. Even the Palo Alto support staff weren’t able to troubleshoot it despite providing a new switch. We left our default Juniper switch as our only line of defense because we only had like 4 VMs on our physical network. Unfortunately, an inject wanted us to compare and contrast different vendors (Cisco vs Palo Alto). For attempted support, we tipped in favor of Palo Alto as Cisco had sent 0 people.

At the end, we were all-hands on-deck writing the rest of the injects since our entire network was basically locked down. Nothing was too crazy, but it was hard to focus because all our brains were fried from handling so many curve balls. Despite our exhaustion, we really wanted the competition to keep going because we had all our services up and needed to close the gap, but there’s only so much time in a day.

Post-Competition

My team and I composed ourselves in my room as we went over our combined experiences to decompress from the competition. Based on the preliminary public scoreboard, we knew top 4 was Stanford and UVA firmly in top 2, then DePaul University followed by us for 3rd and 4th. I tried to keep a positive spin on things. We were the only team to get the active incident, so that should close the gap between us and DePaul, even if we had no chance to catch up with the other 2. We were probably fighting for 3rd with the possibility of being in the lead depending on how much the active incident was worth (we didn’t know at the time). It was a bitter pill to swallow but considering our odds and our lack of nationals experience, we agreed that getting 3rd would be a huge accomplishment in the eyes of many. It was 3rd or bust.

After the brief resting period, it was time for the networking and recruiting events for all the sponsors. All the representatives from sponsors were a pleasure to talk to (and a lot of really cool opportunities offered), but I think the highlight of the night was talking to some of the other competitors and especially to our red team. Despite their reputation, the red team are all very friendly. The red team revealed themselves to be the red team captain, syndrowm, and Alex. We talked about the competition, and they gave us some insight into what it was like to be on the red team and some of their experiences in the industry. Huge shoutout to them for really pushing us in the competition, and I hope to see them again in the future.

By the time the event was closing down, we were all completely exhausted. We had been up since 8AM, and it was now 10PM. Of those 14 hours, we had been competing for more than 8. We were all hungry, but too tired to go out and get food. We decided to UberEats food from a local restaurant.

The food wasn’t great, but at least it was something. In a way, I guess we were hoping for something similar during the awards ceremony. After hours of talk about improvements for next year and whether or not we would compete in the next day’s event, Panoply. Panoply is a king-of-the-hill competition, where there’s basically like 50+ shared boxes that are all vulnerable and require teams to pwn them and write their flag in certain places depending on the scored service. None of us had done it before, so a few members didn’t want to wake up early just to lose. Eventually, though, we decided that we would compete in Panoply. We had nothing to lose, and we were already here. It was best to make the most of our time at nationals, especially since some of us wouldn’t get another chance.

And so, we said our goodnights and went to bed.

Four Hundred // AWARDS

Waking up at 8AM was hard. So hard that some members called it quits and went back to sleep (they were feeling sick, it’s just an inside joke). But nonetheless we were a few members down, and we felt that at breakfast. We were all pretty quiet, pretty tired, and all pretty sure we were going to lose Panoply, but we were going to try anyways.

When we got to the competition room, it was set up in a dimly lit, open ballroom with several circle tables for teams to play at. We just picked the closest one and tried to figure out what was going on. The instruction sheet on the table served very useful for our first purple team event. For NCCDC’s Panoply, the host an internal website that lists the scoreboard, targets, etc. of the competition, but the target list and the actual list of running systems weren’t exactly 1-1. Eventually, we found some hosts with some low hanging fruit (anonymous SMB write) and just started going from there. There were a couple of vulnerabilities I exploited that I picked up from CPTC last year, but for the most part, it was just a lot of enumeration and trying to find a public POC. The event was only 2 hours, so we didn’t have a lot of time to pwn more than a small handful of systems. We ended up getting 3rd place, which was a lot better than we expected. We were all pretty happy with the results, but we didn’t think there’d be much of a reward for 3rd place in a side event, so it wasn’t anything to write home about.

After one of the lunches of all time, it was time for the real awards ceremony.

We filed into the big ballroom that we had been in for the in-brief for the past 2 days. This time though it was packed. There were many visitors, staff, etc. all packed into the rows of seats. We sat down in the middle, right to the camera crew. Coincidentally, we also sat next to one of our red teamers. Now was the time to see if all our hard work had paid off: 3rd or bust. After a ton of speeches and raffles, it was finally time for the awards section. First, they ended up announcing Panoply winners. Since they started off with third, I was pretty surprised to hear my team’s name called out for an award, but it was really cool to see that the top 3 were all recognized. I went up to the stage and got the red trophy. As I was walking up, I thought to myself, “If we don’t get 3rd place overall, this will be the only trophy we get.” After some brief pictures, I went back down clutching the trophy. The other Panoply winners were called up, and then it was time for the final results.

Suddenly, it was just like a month prior. “In 3rd place,” I held my breath. “…DePaul University!”

My heart sank. “If we weren’t third, then… we didn’t make it. There’s no way we could’ve made it as a first-year team anyways. I shouldn’t have gotten everyone’s hopes up for nothing. We made it to nationals, at least? I should be happy.”

Cheers and applause echoed throughout the room as the 3rd place team rightfully marched up the stage and were awarded a trophy for all their efforts. My mind began to fog as I the past 3 years of “if-onlys” and “almosts” began to flood my mind. A faint “and in 2nd place” floated through my ears as I continued to stare a hole into the floor.

“…Cal Poly Pomona!”

“Congrat- huh? Did he say ‘Cal’? Weird, that’s not how you spell UVA…”

And then it finally dawned on me. I looked to my left; the whole team was there beside me in similar disbelief. In that moment, all we could muster was a collective, “what?” before standing up. My legs shook as I wobbled up to the stage, and I could feel my heart beating out of my chest. I was in complete shock. I couldn’t believe it. I couldn’t believe we had done it. Even holding the trophy with everyone on stage, I couldn’t believe we placed 2nd. 2009 was the last time CPP placed in the top 3. CPP didn’t place in the top 3 for 13 years, and the CPP CCDC team hadn’t been to nationals since 2016. 2 years ago, the team failed to make it to regionals. 400 days ago, the team got last place. In spite of it all, this time we did not fall short.

Stanford would end up getting a well-deserved first place, but with the west coast sweep, we were all ecstatic. The wet season had come, and it was raining W’s.

Making The Streak

After 3 long years of CCDC, I’m finally ready to retire. Since I’m graduating a semester early, though, it’s not like I could compete again anyways. I’m glad I was able to end my CCDC career on a high note. I know we could have swept the whole competition with our technical knowledge in retrospect, but experience is invaluable and now they have it in spades. I could not be prouder of my team. They’ve come so far in such a short amount of time, and they’re licking their plates, hungry for first. But the drought is over, and it looks like wet season is here to stay. To my team, if you’re reading this, I’ll be waiting for the good news on the other side of the competition.

Postscript

Tips

1. Treat customer interactions seriously. Good vs great customer service can be the difference of like 5% of the overall score.
2. Everyone needs to know networking. It’s the foundation of everything.
3. Everyone needs to be able to flex into business. This means being able to write professionally, speak professionally, and be able to do it on the fly.
4. Communication is key. Write things down because you will forget.
5. The red team isn’t as scary as you think. Stay focused and you’ll cut them off eventually.
6. Be familiar with how to use your systems and how they work. If you don’t know how it works, you won’t know how to secure it when you’re given a wonky environment.
7. Preparation makes all the difference. Have templates, pre-written injects, gameplans, etc.
8. Competition support staff are your friends. Treat them with respect and they will help you if you ask (you should).
9. You will learn the most if you try to do things “the right way” with some liberties. You’re trying to be a cyber professional, so use this as an opportunity to practice. Just don’t be too rigid and realize CCDC isn’t 100% real life either.
10. Have fun. It’s a competition, but it’s also a learning experience. You’re not going to win every time, so make the most of it.

Other blog posts

• Tanay’s blog post about nationals
• NCCDC Red Team’s 2023 blog post