How I Think GIAC Can Improve CyberLive Exams
Earlier this year I completed the SANS Undergraduate Certificate in Applied CyberSecurity program, during which I took four GIAC certification exams, GFACT, GSEC, GCIH, and GPEN. Two of those exams (GCIH/GPEN) are in GIAC’s relatively new CyberLive format. CyberLive is GIACs step into the ‘practical’ exam space as, along with multiple-choice questions, the exam contains built-in virtual machines students must use to answer questions. As a huge fan of practical exams, I’ve spent a long time thinking about how those exams can improve. Of course, this is my opinion, but I wanted to give some feedback. Below I’ll outline my general feedback, and what I think could be done to improve these exams. I hope it will provide at least some sort of value.
The Problem With Multiple Choice Exams
In today’s world of cybersecurity certifications, multiple-choice exams are becoming less and less relevant. Companies like Offensive Security, eLearnSecurity, TCM Security, and more are leading the charge in their fully practical exams. Not only is this reshaping the certification landscape but it is also dramatically changing the public’s opinion on certifications. The general cybersecurity population is much more likely to respect hands-on/practical exams than fully multiple-choice, and for good reason. Multiple-choice exams are easy to cheat on, easy to memorize, and have a hard time proving actual competency in the studied topic. It’s because of this that I have been very vocal about the need for GIAC exams to change the way they test their students. On some of the more low-difficulty exams like GSIF, multiple-choice works as they are more theory than practice, but once a student moves into 500+ level courses the amount of labs-to-book material begins to shift wildly. The more lab-based material a course has, the harder it is for a multiple-choice exam be a proficient means of testing a student’s level of understanding. These statements must be, at least partially, believed and supported in the GIAC organization as they are attempting to address this issue with the CyberLive program. And, as much as I am excited for this program to develop, I personally feel like the exams I have taken so far leave much to be desired.
The CyberLive Program
Going through the ACS program I have experienced two of the five available CyberLive exams; and continuing into the BACS program I will be taking yet another two — GCIA and GCFA. Because of this I feel like I have at least some ground to stand on when giving feedback on how the exams are being structured. Specifically I will have to focus on more Offensive topics as both GCIH and GPEN are offensive, rather than the more defensive GCIA/GCFA CyberLive exams. Both GCIH/GPEN structured their exams in a ~80/20 split of multiple-choice to hands-on questions. The problem with this is that, although there is a practical portion, a student could theoretically miss most or all of the practical questions and still pass the exam — if their multiple-choice approach was perfect. Which still gives wiggle room to corrupt the image of the certification. Further than that, both GCIH and GPEN are 500 level PenTesting certifications (GCIH kind of totes the line but is still seen as a Red cert). This means that potential students are going to be weighing these certifications against other well-known PenTesting certs. Specifically certs like OSCP, eCPPT, PNTP, CRTP, etc. In his 2019 video Top 3 Certifications for Landing an Ethical Hacking Job, Heath Adams (aka The Cyber Mentor) placed GPEN as #3 behind #2 eCPPT, and #1 OSCP. He placed it there for a couple reasons but he specifically talks about the lack of practicality in the exam compared to his #2/#1 picks. As you may well know OSCP is arguably the most famous PenTesting certifications because of its grueling 24/hour fully-practical exam. It is the epitome of putting your money where your mouth is in the hacking world. The up-and-coming eCPPT cert from eLearnSecurity is also making waves with its 7-day fully practical format. And TCM Security’s PNPT is absolutely killing it with its fully practical exam. The difference that kicks GPEN below these competitors is not the quality of the training, but the approach to verifying a student’s comprehension of the test material. In fact in his video Heath points out how impressive the syllabus is! This discrepancy in test quality to material quality is a common response I have seen in other discussion forms as well, although this is anecdotal evidence. It seems that the way students are being tested is just as important as the way they are being taught; and in the world of practical cybersecurity teaching, practical cybersecurity testing is proving to be imperative.
My Idea For A Better Exam
So, following my father’s ever present advice of “Don’t complain unless you have a solution”, I wanted to lay out what I think CyberLive should look like. In a high-picture view I think the CyberLive exams should mirror closely to the eLearnSecurity Junior Penetration Tester certification (I currently hold this certification and so have experienced this exam….twice). First let me explain their exam (disclaimer this was in 2020):
When students are ready to attempt the eJPT they navigate to the Exams tab in their student portal. They click on a Start Exam button and are immediately given VPN credentials, a Rules of Engagement PDF, and access to the test questions. Breaking this down, students use their own machine (whether local machine or a Kali Linux VM) to VPN into the test network, they’re told what they can and cannot do while connected to that network, and have a list of questions they must answer. Everything from that point is up to them to figure out. For instance, remember this is a PenTesting exam, one question was similar to “What is the CEO’s Email” — this is an extremely open ended question. I was unable to answer this question until after I had compromised a machine that contained information about employees and saw the CEO’s email listed. That was quite an uphill battle to answer one question, which the exam followed up with 19 more. They were in fact multiple-choice questions but the only way to have any idea what the answer could be was to compromise the network and enumerate. Students had 72-hours of access to the test environment and to the question page.
This is what I believe CyberLive should model. Not to the exact letter but with a similar feel. GIAC differs from eLearnSecurity as GIAC exams must be proctored, and I would like to keep this that way. Retaining a proctored exam means that there is no viable way to run the test for 72-hours. But that is okay! CyberLive exams should still fall in the 3–5 hour time limits as they are right now, it’s just the amount of questions, and the environment change. The way the questions are asked should be changed as well. Instead of extremely generic questions like eJPT, GIAC should retain specificity, pertaining to specific tools and methodologies which gives students the ability to discern what knowledge they’re being tested on during that question. This would allow students to understand what they need to do without guessing as much as I did during eJPT. This would be the way that GIAC can retain the current time limit, because students wouldn’t have to guess what they need to do, just how they need to do it.
Painting A Picture
A student completes the SEC560: Network Penetration Testing and Ethical Hacking SANS course. They feel confident in the material and schedule their proctored GIAC exam. Exam day comes and they run all proper procedures, disconnecting any additional monitors, disabling any unauthorized services/software, have only written notes and course books, etc. The proctor verifies everything is correct and begins the test. Once the exam is brought up students view the questions they need to answer and open up the exam VM. Tools needed to complete any/all tasks required during the exam are pre-installed and optimized on the test machine and there is a ReadMe file on the desktop if the student forgets where a tool is located in the file system — no other information than that.
Question 1: “Using Nmap, on which IP is FTP running, and what version is it?” — below the question are 10 different choices, there are a total of 5 unique IPs contained in the options listed but every FTP version is unique. This would mean that if the student had no idea and guessed they’d be left with a 10% chance of being right. That would make most people more willing to figure it out.
This, in my opinion, is perfect. The student automatically knows that they are in the Scanning and Enumeration phase, they need to use Nmap, and further than that they need to run Nmap in a way that doesn’t just find what service is running but to find the specific version as well. A student would need to understand how to run Nmap in this way, however basic this is, and perform it to find the correct answer. This is the same thing as a “Read this Nmap command” but instead of knowledge regurgitation it’s knowledge application.
Question 2: “Using Metasploit OR Nmap, attempt to brute force the FTP server. What is the password discovered?” — below the question are 15 different choices.
This is a great way of making the student stop and think about what they need to do, while giving them enough information to keep them from freezing. The student can either rely on a Metasploit module, or a NSE script to find the correct answer. They’re given the freedom to choose what tool they’re more comfortable with, but still enough guidance to keep their pace up to finish the test within the time limits.
Questions can get more in-depth, involving more and more tools and techniques as they further penetrate the exam network. Students can be required to pivot and perform exfiltration of data to retrieve vital insight to answer questions. The exam could begin to feel like the Day 6 CTF in SEC560. Each question hinting, or flat out saying, what tool to use and what task to perform, but still leaving it up to the student to perform the action.
Once all questions have been answered within the test environment the student submits the test and retrieves their passing/failing results.
Conclusion
I know this isn’t perfect. It adds in even more potential testing pains from troubles with the VM, to troubles with freezes/resets, and even more problems with the exam network. Proctors must wield more responsibilities, and students have to complete more pre-testing steps. But the point still remains, fully-practical exams are the way forward even with testing pains. I know plenty of people who had machine crashes during eCPPT and OSCP but not a single one complained after passing, and are quite proud now that it is over.
I absolutely love my GIAC certifications, and am ecstatic to continue my journey with SANS/GIAC, and it’s for that reason that I’d love to see improvements. GPEN deserves that #1 spot in any future Top 3 videos that inevitably come out, but it won’t as long as it still relies on a multiple-choice backbone. I hope these thoughts are of some sort of value-add and look forward to the future of CyberLive.
