Finding bugs and vulnerabilities is an important part of programming and system administration to maintain proper functionality and prevent failure. Fuzzing has become a popular method for finding bugs since it delivers good results. In detail, it utilizes relatively few resources and time by leveraging randomness to test a large part of the code for unknown bugs. Although fuzzing has a high demand, the evaluation of fuzzers on real-world bugs is sparse. Moreover, the few fuzzing benchmarks on real programs fail to compare the performance of fuzzers on new CVEs. Another problem is that current benchmarking tools do not offer the ability to easily customize and expand the set of test cases. To solve these issues, this work proposes an extendable benchmarking tool on real-world bugs. Our approach consists of the expandable benchmarking tool Pineapple, which focuses on testing fuzzers at finding CVEs. Our tool matches the results of the fuzzers against the proof-of-concept file of the CVE automatically by using stack traces to check if the bug has been found. Furthermore, Pineapple’s benchmarking process can be fully customized with JSON config files. This allows us to provide ease of use as well as expandability for Pineapple to be adaptable for new test cases and fuzzers to assist with additional data capturing. Utilizing Pineapple, this work evaluates the capabilities of modern fuzzers at detecting bugs that have been published since 2019. In 7 runs, this benchmark was able to measure a high consistency of finding recently detected bugs within 24 hours for only one of three test cases. The benchmarking tool Pineapple has been published on GitHub to encourage others can compare more fuzzers on their test cases.