Malware Analysis with ML
Every day, security researchers encounter hundreds of thousands of new malicious files. Traditional antivirus engines, built on hand-crafted signatures, are simply overwhelmed by the volume and variety of modern malware. Machine learning has emerged as the essential tool for automating detection, classification, and deep analysis of malicious code at scale — but using it correctly requires understanding both the techniques and their limits.
The failure of signature-based antivirus
For decades, antivirus software worked by comparing files against a database of known malicious signatures — cryptographic hashes or byte patterns unique to previously identified malware. This approach was highly effective when the number of malware variants was small and threat actors lacked the tools to rapidly mutate their code. That world no longer exists.
Modern malware authors use polymorphic and metamorphic engines that automatically rewrite code on each infection, producing millions of functionally identical but byte-distinct variants. A signature written for one variant will miss every subsequent mutation. The AV-TEST Institute registers over 450,000 new malicious programs every single day — a volume no team of human analysts could process with manual signatures alone.
The deeper problem is that signature-based detection is inherently reactive. A threat cannot be added to the database until it has already been observed in the wild, analyzed, and published. That window between first appearance and signature deployment — sometimes days, sometimes weeks — is exactly when attackers do their most damage. Machine learning changes the game by enabling generalization: a model trained on malware characteristics can detect novel variants it has never seen before, based on patterns that persist across mutations.
Signature-based detection asks: "Have I seen this exact file before?" Machine learning asks: "Does this file look like malware?" The second question is far more powerful against novel and mutated threats — but it introduces new failure modes, particularly false positives and adversarial manipulation.
Static analysis: reading the binary without running it
Static analysis examines a malware sample without executing it. For ML-based approaches, this means extracting features from the file's raw bytes, structure, and metadata — then feeding those features into a classifier. The advantage is speed and safety: no sandboxed execution environment is required, and analysis can be completed in milliseconds.
PE header features
Most Windows malware is packaged as Portable Executable (PE) files. The PE header contains rich metadata: the number and names of imported DLLs and functions, section names and entropy levels, compilation timestamps, resource tables, and subsystem flags. Malware tends to exhibit distinctive patterns in these fields — unusually high entropy in code sections (indicating packing or encryption), imports of low-level Windows API functions associated with injection or persistence, and anomalous section sizes.
Import address table and API call patterns
The set of Windows API functions a binary imports is one of the strongest static signals for malware classification. Ransomware tends to import cryptographic APIs. Spyware imports screenshot and keylogging APIs. Rootkits import kernel-level functions for hiding processes and files. A trained classifier can map API import sets to malware families with high accuracy, even when code has been substantially modified.
Byte n-grams and raw feature extraction
At the most fundamental level, malware detection can be framed as a sequence classification problem over raw bytes. Byte n-grams — sequences of n consecutive bytes extracted from the binary — capture local structural patterns without requiring semantic understanding of the code. Early ML malware classifiers used n-gram frequency histograms as features fed into naive Bayes or SVM classifiers. While these approaches are now supplemented by deeper techniques, they remain useful for rapid triage at high throughput.
Dynamic analysis: watching malware execute
Static analysis is blind to runtime behavior. A heavily obfuscated dropper may look clean statically while unpacking malicious code in memory and calling destructive APIs only during execution. Dynamic analysis addresses this by running the sample in a controlled sandbox environment and recording what it actually does.
Modern sandboxes instrument the operating system to capture: system calls and their arguments, network connections and DNS requests, file system operations (reads, writes, deletions, encryptions), registry modifications, process creation and injection events, and memory allocation patterns. This behavioral trace is then fed into ML models that classify the sample based on what it did, not what it looks like statically.
Static analysis is fast but can be defeated by obfuscation. Dynamic analysis sees through obfuscation but is slower, requires execution infrastructure, and can be defeated by sandbox-aware malware that detects the analysis environment and alters its behavior. Production ML pipelines typically chain both: static classifiers triage at high speed, flagging suspicious samples for deeper dynamic analysis.
Feature extraction and supervised classification
The most common ML architecture for malware classification is supervised learning: a model trained on labeled samples (known malware vs. benign, or labeled by family) learns to map features to classes. The practical challenges lie in feature engineering and label quality.
Deep learning for obfuscated and packed malware
Obfuscation is the primary technique attackers use to defeat static ML classifiers. A packer wraps the original malicious code in a layer of decryption logic; the outer shell looks benign or at least unfamiliar, while the actual payload only appears in memory during execution. Metamorphic engines rewrite code structure — replacing instructions with semantically equivalent alternatives, shuffling code blocks, inserting junk instructions — while preserving functionality.
Deep learning approaches address obfuscation in several ways. Recurrent neural networks (RNNs) and transformers applied to byte sequences can learn long-range dependencies that persist even through reordering and insertion. Disassembly-level analysis extracts the actual instruction stream rather than raw bytes, making simple obfuscations transparent. Code normalization pipelines attempt to canonicalize equivalent code representations before feature extraction, reducing the surface area that obfuscation can exploit.
The most powerful recent development is applying large language models to disassembled code. Treating assembly instructions as tokens, LLMs pretrained on large corpora of benign and malicious code can develop semantic understanding of what code does rather than merely what it looks like — enabling detection of novel obfuscation techniques that defeat pattern-based approaches.
Graph neural networks for malware relationship mapping
Individual malware samples do not exist in isolation. They share code, infrastructure, and development patterns with other samples from the same threat actor or malware family. Graph neural networks (GNNs) exploit these relationships for both classification and attribution.
Code similarity graphs connect samples that share code segments, enabling propagation of labels from known samples to unknown variants. Call graphs represent the internal function call structure of a binary as a graph, and GNNs can learn family-distinctive graph topologies that persist even when individual functions are obfuscated. Infrastructure graphs connect samples through shared C2 domains, IP addresses, SSL certificates, and WHOIS records — revealing threat actor infrastructure even when code is completely rewritten.
Graph-based analysis is particularly powerful for threat attribution because attackers find it much harder to change their infrastructure patterns than their code. Even if a malware author completely rewrites their payload, they often reuse hosting providers, domain registrars, or SSL certificate patterns — connections that GNN-based threat intelligence platforms can detect and link back to known threat actors.
Behavioral analysis and sandboxing at scale
Modern commercial sandboxes — such as Cuckoo, ANY.RUN, and enterprise platforms from vendors like CrowdStrike and Palo Alto — run thousands of samples per hour in instrumented virtual machines, collecting behavioral telemetry that feeds directly into ML pipelines. The ML component handles the classification and prioritization: not every sandboxed sample deserves the same level of analyst attention, and ML-driven scoring helps SOC teams focus human expertise where it matters most.
Key behavioral features include API call sequences (modeled as time series or text), network indicators (C2 beaconing patterns, DNS-over-HTTPS usage, protocol anomalies), filesystem operations (mass encryption indicating ransomware, shadow copy deletion, backup removal), and process injection techniques (DLL injection, process hollowing, APC injection detected through memory introspection).
Adversarial examples: when attackers fool the classifier
ML-based malware classifiers are themselves vulnerable to attack. An adversarial example in the malware context is a malicious file that has been minimally modified to cross the decision boundary of a classifier — appearing benign to the model while retaining full malicious functionality.
Feature manipulation attacks add benign-looking bytes or metadata to a malicious binary — appending benign code to lower the model's malice score without affecting execution. Gradient-based attacks query the model repeatedly to map its decision boundary, then craft minimal perturbations that flip the classification. Semantic-preserving transformations apply code transformations that a metamorphic engine can already perform — inserting equivalent instructions, reordering independent code blocks — but directed specifically at evading a target classifier.
This is not theoretical: researchers have demonstrated adversarial attacks against commercial antivirus ML engines in both white-box and black-box settings. Any production ML malware detection system must assume that sophisticated attackers will probe and attempt to evade it.
Threat intelligence integration
ML-based malware analysis does not operate in isolation from broader threat intelligence. The most effective platforms integrate classifier output with structured threat intelligence feeds — MITRE ATT&CK technique tagging, YARA rule matching, known threat actor TTPs, and vulnerability intelligence — to produce actionable context alongside detection verdicts.
When a ML classifier flags a sample as likely ransomware belonging to the LockBit family, that verdict should automatically trigger enrichment: known LockBit C2 infrastructure to block, specific persistence mechanisms to hunt for, known exfiltration destinations to monitor. The combination of ML speed and human-curated threat intelligence depth is what makes modern security platforms effective at scale.
The real value of ML in malware analysis is triage at scale: processing hundreds of thousands of samples per day, prioritizing which ones deserve human attention, and clustering unknown variants around known families. No team of human analysts could cover this volume manually. ML does not replace the skilled reverse engineer who deeply analyzes a novel threat — it creates the space for that expertise to be applied where it matters.
Next
Module 5 examines how AI is transforming vulnerability assessment and penetration testing — from smart fuzzing and AI-assisted code review to automated CVE prioritization and AI-powered red team operations.