Help for package CBTF

Type:

Package

Title:

Caught by the Fuzz! - A Minimalistic Fuzz-Test Runner

Version:

0.4.0

Date:

2025-07-31

Maintainer:

Marco Colombo <mar.colombo13@gmail.com>

Description:

A simple runner for fuzz-testing functions in an R package's public interface. Fuzz testing helps identify functions lacking sufficient argument validation, and uncovers problematic inputs that, while valid by function signature, may cause issues within the function body.

URL:

https://mcol.github.io/caught-by-the-fuzz/

BugReports:

https://github.com/mcol/caught-by-the-fuzz/issues

Imports:

cli (≥ 3.6.5)

Suggests:

testthat (≥ 3.2.3), withr (≥ 3.0.2)

License:

GPL-3

Encoding:

UTF-8

RoxygenNote:

7.3.2

Config/testthat/edition:

NeedsCompilation:

Packaged:

2025-07-31 13:07:36 UTC; marco

Author:

Marco Colombo

[aut, cre]

Repository:

CRAN

Date/Publication:

2025-08-19 14:50:31 UTC

CBTF: Caught by the Fuzz! A minimalistic fuzz-test runner

Description

This package implements a very simple mechanism for fuzz-testing functions in the public interface of an R package.

Details

Fuzz testing helps identify functions lacking sufficient argument validation, and uncovers sets of inputs that, while valid by function signature, may cause issues within the function body.

The core functionality of the package is fuzz, whose aim is to call each provided function with a certain input and record the output produced. If an error is generated, this is captured and reported to the user, unless the error message matches a pattern of whitelisted errors. The objects returned by fuzz can be printed with print.cbtf and summary.cbtf.

The helper function get_exported_functions identifies the functions in the public interface of a given package, facilitating the generation of the list of functions to be fuzzed.

Function test_inputs by default generates a large set of potentially problematic inputs, but they can be limited just to the desired classes of inputs.

Author(s)

Maintainer: Marco Colombo mar.colombo13@gmail.com (ORCID)

Fuzz-test the specified functions

Description

This function calls each of the functions in funs with each of the objects specified in what, recording if any errors or warnings are thrown in the process.

Usage

fuzz(
  funs,
  what = test_inputs(),
  package = NULL,
  listify_what = FALSE,
  ignore_patterns = "",
  ignore_warnings = FALSE
)

Arguments

funs

A character vector of function names to test. If a "package" attribute is set and is no package argument is provided, functions are loaded from the namespace specified in the attribute.

what

A list of objects to be passed, one at a time, as the first argument to each function in funs. Ideally, the list should be named, so that each input tested can be pretty-printed with the corresponding name. For unnamed lists, a deparsed representation of the inputs will be used, which may appear unwieldy in some cases. If nothing is provided, a default set of inputs generated by test_inputs will be used.

package

A character string specifying the name of the package to search for functions. If NULL (default), the function will first check the "package" attribute of funs, and if that is not set, names will be searched in the global namespace.

listify_what

Whether each input in what should also be tested in its listified version (FALSE by default). When set to TRUE, if what is list(x = x), the function will operate as if what were list(x = x, "list(x)" = list(x)), for any input object x.

ignore_patterns

One or more strings containing regular expressions to match the errors to ignore. The string "is missing, with no default" is always ignored.

ignore_warnings

Whether warnings should be ignored (FALSE by default).

Details

In order to reduce the number of false positive results produced, this function applies the following set rules, to establish if an error or warning condition should ignored (whitelisting):

If the name of the function appears in the error or warning message, as it is considered that the condition has been handled by the developer.
If the error or warning message contains the text "is missing, with no default", which is produced when a missing argument is used without a value being assigned to it.
If the error or warning message contains any of the patterns specified in ignore_patterns.
If a warning is thrown but ignore_warnings = TRUE is set.

In all whitelisted cases, the result is "OK", and the message that was received is stored in the ⁠$msg⁠ field (see the Value section).

Value

An object of class cbtf that stores the results obtained for each of the functions tested. This contains the following fields:

$runs

a list of data frames, each containing the results of fuzzing all the functions in funs with one of the inputs in what, with attribute "what" containing . The data frame contains the following columns and attributes:
- fun: The name of the function tested.
- res: The result of the fuzz test, see below for the possible values.
- msg: The error or warning message returned by the function, if any.
- ⁠attr(*, "what")⁠: The character representation of the input tested.

$package

a character string specifying the package name where function names were searched, or NA if none was provided.

The res column in each of the data frames in the ⁠$runs⁠ field can contain the following values:

OK: either no error or warning was produced (in which case, the msg entry is left blank), or it was whitelisted (in which case, the message received is stored in msg).
SKIP: no test was run, either because the given name cannot be found, or it doesn't correspond to a function, or the function accepts no arguments, or the function contains a call to readline; the exact reason is given in msg.
WARN: a warning was thrown for which no whitelisting occurred and ignore_warnings = FALSE; its message is stored in msg.
FAIL: an error was thrown for which no whitelisting occurred; its message is stored in msg.

Examples

## this should produce no errors
res <- fuzz(funs = c("list", "matrix", "mean"),
            what = test_inputs(c("numeric", "raw")))
summary(res)

## display all results even for successful tests
print(res, show_all = TRUE)

## this will catch an error (false positive)
fuzz(funs = "matrix",  what = test_inputs("scalar"))

Get the names of the exported functions of a package

Description

This function extracts the exports from the namespace of the given package via getNamespaceExports and discards non-fuzzable objects (non-functions and functions with no arguments). The set of names returned can be further restricted via the ignore_names argument.

Usage

get_exported_functions(package, ignore_names = "")

Arguments

package

Name of the package to fuzz-test.

ignore_names

Names of functions to ignore: these are removed from the names returned. This can be helpful, for example, to discard function aliases.

Value

A character vector of the names of the fuzzable functions exported from the given package, with the "package" attribute set. This can be used directly as the funs argument of fuzz without need to specify the package argument.

Examples

## get the fuzzable functions in the public interface of this package
funs <- get_exported_functions("CBTF")

Compute the number of inputs tested

Description

Compute the number of inputs tested

Usage

## S3 method for class 'cbtf'
length(x)

Arguments

x

An object of class cbtf.

Value

An integer corresponding to the number of inputs tested in a run.

Examples

res <- fuzz(funs = c("list", "matrix", "mean"),
            what = test_inputs(c("numeric", "raw")))
length(res)

Print the results from a fuzz run

Description

This formats with colours the results from a fuzz run and prints them to the terminal.

Usage

## S3 method for class 'cbtf'
print(x, show_all = FALSE, ...)

Arguments

x

An object of class cbtf.

show_all

Whether all results should be printed. By default (FALSE), only the functions that reported an error or a warning are printed. If TRUE, all functions tested are printed, including those that were successful or were skipped.

...

Further arguments passed to or from other methods. These are currently ignored.

Value

No return value, called for side effects.

Examples

res <- fuzz(funs = c("list", "matrix", "mean"),
            what = test_inputs(c("numeric", "raw")))
print(res, show.all = TRUE)

Results summary from a fuzz run

Description

Reports some summary statistics from the results of a run of fuzz.

Usage

## S3 method for class 'cbtf'
summary(object, ...)

Arguments

object

An object of class cbtf.

...

Further arguments passed to or from other methods. These are currently ignored.

Value

A data frame containing the following columns and attributes is returned invisibly:

fun

The names of the function tested.

what

The inputs tested.

res

One of "OK", "FAIL", "WARN" or "SKIP" for each combination of function and input tested (see the Value section in fuzz).

msg

The message received in case of error, warning or skip, or an empty string if no failure occurred.

attr(*, "summary_table")

The tabulation of results that was printed out.

Examples

res <- fuzz(funs = c("list", "matrix", "mean"),
            what = test_inputs(c("numeric", "raw")))
summary(res)

Default input tests

Description

This function provides a selection of potentially problematic inputs by class. List inputs are very limited by design, as they can be automatically generated by setting listify_what = TRUE in fuzz.

Usage

test_inputs(use = "all", skip = "")

Arguments

use

Names of input classes to use. Valid names are "all" (default), "scalar", "numeric", "integer", "logical", "character", "factor", "data.frame", "matrix", "array", "date", "raw" and "list".

skip

Names of input classes to skip.

Value

A named list of inputs corresponding to the input classes selected.

Examples

## only the scalar and numeric tests
inputs1 <- test_inputs(c("scalar", "numeric"))

## everything but the data, raw and list tests
inputs2 <- test_inputs("all", skip = c("date", "raw", "list"))

CBTF: Caught by the Fuzz! A minimalistic fuzz-test runner

Description

Details

Author(s)

See Also

Fuzz-test the specified functions

Description

Usage

Arguments

Details

Value

See Also

Examples

Get the names of the exported functions of a package

Description

Usage

Arguments

Value

See Also

Examples

Compute the number of inputs tested

Description

Usage

Arguments

Value

Examples

Print the results from a fuzz run

Description

Usage

Arguments

Value

See Also

Examples

Results summary from a fuzz run

Description

Usage

Arguments

Value

See Also

Examples

Default input tests

Description

Usage

Arguments

Value

See Also

Examples