High system call latency and "scap" file bloat during external script injection

Communities General
1

Hi everyone, I’ve been using Sysdig OSS to perform some forensic analysis on a staging environment where we test various automation scripts. I’ve run into a specific issue where the .scap files become massive—much larger than usual—whenever I’m running tests that involve external executors or high-frequency script hooks.

I actually started digging into this after spending some time on deltaexector com to see how their iOS-specific framework manages process-level injections. I’ve noticed that when I try to monitor similar script behaviors, the proc.name in my sysdig output shows thousands of tiny read and write events that seem to trigger a loop. I was wondering if the deltaexector com approach to bypassing certain sandboxing might be causing these “noise” events that are flooding my sysdig capture.

Has anyone else seen a related issue where using the sysdig -p "%proc.name %evt.type %evt.args" filter results in significant CPU spikes when an external script is active? I’m also having trouble with the Falco engine triggering “Sensitive file opened for reading” alerts every time these executors refresh their background processes. Is there a way to fine-tune the Sysdig filter to ignore these specific thread IDs without losing visibility on the actual container performance? I’d really appreciate any tips on how to keep the forensic captures manageable without the UI freezing up during these high-activity bursts.

1 Like
2

Great question!

I’ll get someone on our team to look at this and respond.

3

Hi

I’m not familiar with deltaexcator, but what you describe should be expected with programs that rely heavily on I/O.

The behavior you’re seeing is expected when capturing without syscall filters — read, write, close, polland similar I/O syscalls typically account for 90%+ of all system call volume on any active system. When your automation scripts run, they likely generate thousands of these events per second (pipe I/O, file reads, socket writes), and capturing all of them is what causes the .scap file bloat, CPU spikes, and UI freezes.

Below are some suggestions that may help you to work around the issues you’re facing.

1. Filter out noisy I/O syscalls during capture

Apply a filter **at capture time** so the events never hit disk:

```bash

# Exclude the noisiest I/O syscalls

sudo sysdig “not evt.type in (‘read’,‘write’,‘close’,‘poll’,‘select’,‘epoll_wait’,‘recvfrom’,‘sendto’)” -w capture.scap

# Or, if you only care about process execution and file opens:

sudo sysdig “evt.type in (‘execve’,‘openat’,‘connect’,‘accept’,‘unlink’,‘rename’,‘clone’,‘fork’)” -w capture.scap

```

This alone will dramatically reduce your .scap file size (often 10-50x smaller).

2. Limit buffer snaplen and use file rotation

```bash

sudo sysdig \

-s 256 \

-C 5 -W 10 \

“not evt.type in (‘read’,‘write’,‘close’,‘poll’,‘select’,‘epoll_wait’)” \

-w /tmp/trace.scap

```

- `-s 256` — capture only the first 256 bytes of each I/O buffer (default is 80)

- `-C 5 -W 10` — rotate into 5 MB files, keeping the last 10 (ring buffer)

3. Fix the CPU/UI freeze with `-p`

The issue with `sysdig -p “%proc.name %evt.type %evt.args”` is that rendering every event to stdout at high rates overwhelms the terminal. Two options:

- **Write to file instead of stdout**: `sysdig -w capture.scap` and analyze later with `sysdig -r capture.scap -p “…”`

- **Apply the same filters**: `sysdig -p “%proc.name %evt.type %evt.args” “not evt.type in (‘read’,‘write’,‘close’)”`

About the Falco “Sensitive file opened” alerts

This is a separate concern. The executor background process refreshing and reading files like `/etc/shadow` or `/etc/passwd` triggers Falco’s `Read sensitive file untrusted` rule. You can add an exception for the specific process in your Falco rules:

```yaml

- rule: Read sensitive file untrusted

append: true

exceptions:

- name: executor_refresh

fields: [proc.name]

comps: [=]

values:

    - \[your_executor_name\]

```