Remote Execution and Networking¶

The remote-execution layer packages a Python function or script, generates the shell and Python artifacts required to run it elsewhere, transfers those artifacts, fires the job on a target machine, and provides helpers for checking status and downloading results.

It also includes lower-level SSH and networking utilities that show up in remote or multi-machine workflows.

Remote workflow in one view¶

graph LR
    A[RemoteMachineConfig] --> B[RemoteMachine]
    B --> C[JobParams]
    B --> D[FileManager]
    B --> E[generate_scripts()]
    E --> F[submit()]
    F --> G[SSH / transfer]
    G --> H[fire()]
    H --> I[check_job_status()]
    I --> J[download_results()]

Core modules¶

RemoteMachineConfig¶

RemoteMachineConfig is the central description of how a job should run.

Important fields¶

WorkloadParams complements this by describing how a large workload should be sliced into job ranges.

RemoteMachine¶

RemoteMachine accepts either:

• a script path
• an importable callable
• a bound method

and turns that into a runnable remote job.

Typical usage¶

from machineconfig.cluster.remote.models import RemoteMachineConfig
from machineconfig.cluster.remote.remote_machine import RemoteMachine

config = RemoteMachineConfig(
    job_id="nightly-train",
    ssh_host="gpu-box",
    base_dir="~/tmp_results/remote_machines/jobs",
    transfer_method="sftp",
    notify_upon_completion=False,
)

job = RemoteMachine(
    func="src/my_project/jobs/train.py",
    config=config,
    func_kwargs={"epochs": 10},
    data=["data/input.csv"],
)

job.run(run=True, show_scripts=False)

Main methods¶

File layout and locking¶

FileManager gives the remote-job system a stable directory shape. A generated job typically looks like this:

{base_dir}/queued/{job_id}/
├── data/
│   ├── file_manager.json
│   ├── func_kwargs.json
│   ├── job_params.json
│   ├── remote_machine.json
│   └── remote_machine_config.json
├── python/
│   └── cluster_wrap.py
├── shell/
│   └── cluster_script.sh
└── logs/
    ├── status.txt
    ├── start_time.txt
    ├── end_time.txt
    └── results_folder_path.txt

It also manages:

• queue and running-job tracking JSON files
• resource locking for machines with limited concurrency
• history files for completed jobs

SSH helper¶

machineconfig.utils.ssh.SSH is the lower-level transport used by remote jobs and other operational flows.

It can:

• resolve hosts through ~/.ssh/config
• open an SSH connection with Paramiko
• open an SFTP channel when available
• run remote commands
• display local and remote machine specs side-by-side

A common entrypoint is:

from machineconfig.utils.ssh import SSH

ssh = SSH.from_config_file(host="gpu-box")

Networking helper entrypoints¶

Some remote workflows need address discovery rather than full job submission.

helpers_network.address¶

Useful functions include:

• get_public_ip_address()
• get_all_ipv4_addresses()
• select_lan_ipv4(prefer_vpn)

helpers_network.address_switch¶

switch_public_ip_address(max_trials, wait_seconds, target_ip_addresses) automates a public-IP rotation flow using warp-cli and repeated ipinfo checks.

These modules live under scripts/python/helpers because they are also exposed through CLI flows, but they are still useful as library entrypoints.

See also¶

• Sessions for the backend managers that actually host launched jobs
• Layouts for the typed tab and layout schema used by session-oriented flows
• CLI Sessions Reference for the user-facing command layer

API reference¶

Remote models¶

Job parameters¶

File manager¶

Remote machine¶

SSH helper¶

SSH(
    host: str | None,
    username: str | None,
    hostname: str | None,
    ssh_key_path: str | None,
    password: str | None,
    port: int,
    enable_compression: bool,
)

simple_sftp_get(remote_path: str, local_path: Path) -> None

Address helpers¶

select_lan_ipv4(prefer_vpn: bool) -> str | None

Address switching¶