1. Introduction to Malware Analysis

Malware analysis is the process of determining the functionality, origin, and potential impact of malicious software. This critical cybersecurity discipline helps organizations understand threats, develop defenses, and respond to incidents effectively. The following content is what I have been teaching my students in Cyryx College (MSc in Cyber Security)

Types of Malware Analysis

Static Analysis: Examining malware without executing it

• Analyzing file properties and metadata
• Disassembling code
• Examining strings and resources
• Identifying packed or obfuscated code

Dynamic Analysis: Observing malware behavior during execution

• Monitoring system changes
• Tracking network communications
• Analyzing process behavior
• Recording API calls

Hybrid Analysis: Combining static and dynamic techniques for comprehensive understanding

The Analysis Workflow

1. Initial Triage: Quick assessment to determine malware type and priority
2. Static Examination: Deep dive into code structure without execution
3. Dynamic Testing: Controlled execution to observe behavior
4. Memory Analysis: Examining runtime memory artifacts
5. Reporting: Documenting findings and indicators of compromise (IOCs)

2. Setting Up a Safe Analysis Environment

Virtual Machine Configuration

Creating an isolated environment is crucial for safe malware analysis. Here’s how to set up a proper analysis lab:

VMware Workstation Pro Setup:

1. Create a new Windows 10/11 VM with:
   • 4GB RAM minimum
   • 60GB disk space
   • Network adapter set to “Host-only” or “NAT” (never bridged)
   • Snapshot capability enabled
2. Install analysis tools before taking a clean snapshot:
   • Process Monitor (ProcMon)
   • Process Explorer
   • Wireshark
   • x64dbg/OllyDbg
   • IDA Free
   • PEiD or Detect It Easy
   • Autoruns
   • RegShot
3. Configure the VM:
   • Disable Windows Defender and firewall
   • Disable automatic updates
   • Create a non-admin user account for testing
   • Take a “clean” snapshot before any analysis

Network Isolation:

• Use INetSim or FakeNet-NG to simulate internet services
• Configure a separate analysis network segment
• Never analyze malware on production networks

Physical Lab Considerations

For advanced analysis, consider:

• Dedicated analysis machines (air-gapped)
• Hardware write blockers for forensic imaging
• Network tap devices for traffic capture
• Faraday cage for wireless malware analysis

3. Static Analysis Techniques

File Properties and Metadata

Initial File Analysis Steps:

1. Hash Calculation:

Tools: HashMyFiles, CertUtil, or PowerShell
Commands:
- certutil -hashfile sample.exe SHA256
- Get-FileHash sample.exe -Algorithm SHA256

2. File Type Identification:

• Use file headers (magic bytes) to verify true file type
• Tools: TrID, file command (Linux), or hex editor
• Check for common disguises (e.g., .exe renamed to .pdf)

3. Metadata Extraction:

• Examine PE headers for compilation timestamps
• Check digital signatures (often fake or stolen)
• Extract version information and resources

PE File Structure Analysis

Portable Executable (PE) Analysis:

1. PE Header Examination:

• Import Address Table (IAT): Shows API functions used
• Export Address Table (EAT): Functions exported (rare in malware)
• Section headers: Identify code, data, and resource sections
• Entry point: Where execution begins

2. Using PE Analysis Tools:

PEview: Visual PE header explorer
PE Explorer: Commercial tool with advanced features
pestudio: Highlights suspicious indicators

3. Suspicious PE Characteristics:

• Unusual section names (.upx, .nsp, custom names)
• High entropy sections (indicates packing/encryption)
• Mismatched section characteristics
• Suspicious timestamp values

String Analysis

Extracting and Analyzing Strings:

1. Basic String Extraction:

Tools: strings (Linux), Strings (Sysinternals)
Usage: strings -n 8 malware.exe > strings.txt

2. What to Look For:

• URLs and IP addresses
• Registry keys and file paths
• Command and control (C2) domains
• Error messages and debug information
• Cryptocurrency addresses
• Email addresses

3. Encoded String Detection:

• Base64 encoded strings
• XOR encrypted strings
• Custom encoding schemes

Disassembly and Code Analysis

Using IDA Free:

1. Initial Loading:

• Load the executable
• Let IDA perform auto-analysis
• Navigate to entry point

2. Key Analysis Points:

• WinMain or DllMain functions
• String references (Shift+F12)
• Import functions (critical APIs)
• Suspicious code patterns

3. Common Malware Patterns:

; Process injection pattern
OpenProcess
VirtualAllocEx
WriteProcessMemory
CreateRemoteThread

; Persistence pattern
RegCreateKeyEx
RegSetValueEx
"Software\\Microsoft\\Windows\\CurrentVersion\\Run"

4. Dynamic Analysis Techniques

System Monitoring

Process Monitor (ProcMon) Configuration:

1. Filter Setup:

• Process Name contains “malware.exe”
• Exclude normal system activity
• Focus on Registry, File System, and Process events

2. Key Events to Monitor:

• File creation/modification
• Registry key creation/modification
• Process creation
• Network connections (limited in ProcMon)

Process Explorer Usage:

1. Process Tree Analysis:

• Parent-child relationships
• Command line arguments
• Loaded DLLs
• Handle information

2. Verification:

• Check digital signatures
• Verify process paths
• Compare against known good processes

Behavioral Analysis Workflow

Step-by-Step Dynamic Analysis:

1. Pre-execution Preparation:

• Start all monitoring tools
• Clear event logs
• Take VM snapshot
• Prepare network capture

2. Execution Phase:

• Run malware as intended user would
• Document all visible changes
• Allow malware to “phone home”
• Interact with any UI elements

3. Post-execution Analysis:

• Stop all captures
• Export logs and data
• Compare system state
• Identify persistence mechanisms

API Monitoring

Using API Monitor:

1. Configuration:

• Select relevant API categories
• Set breakpoints on critical functions
• Configure call stack capture

2. Critical APIs to Monitor:

File Operations:
- CreateFile, WriteFile, DeleteFile

Process Management:
- CreateProcess, OpenProcess, TerminateProcess

Network:
- WSASocket, connect, send, recv

Registry:
- RegCreateKeyEx, RegSetValueEx

Cryptography:
- CryptEncrypt, CryptDecrypt, CryptHashData

5. Network Traffic Analysis

Wireshark Configuration

Capture Setup:

1. Interface Selection:

• Choose VM network adapter
• Enable promiscuous mode
• Set appropriate capture filters

2. Useful Capture Filters:

# Exclude local traffic
not (src host 127.0.0.1 or dst host 127.0.0.1)

# Focus on specific protocols
tcp.port == 80 or tcp.port == 443

# Capture DNS queries
udp.port == 53

Traffic Pattern Analysis

Identifying C2 Communications:

1. Beaconing Behavior:

• Regular intervals between connections
• Consistent packet sizes
• Specific user agents or headers

2. Data Exfiltration Indicators:

• Large outbound transfers
• Unusual protocols or ports
• Encrypted/encoded payloads

3. DNS Analysis:

• DGA (Domain Generation Algorithm) domains
• DNS tunneling indicators
• Fast flux networks

Protocol Analysis

HTTP/HTTPS Traffic:

1. Request Analysis:

GET /command.php?id=VICTIM001&status=ready HTTP/1.1
Host: malicious-c2.com
User-Agent: CustomBot/1.0

2. Response Examination:

• Command structures
• Downloaded payloads
• Configuration updates

Custom Protocols:

• Identify magic bytes
• Analyze packet structure
• Decode communication format

6. Memory Forensics

Memory Acquisition

Creating Memory Dumps:

1. Tools for Memory Capture:

• DumpIt: Simple command-line tool
• Magnet RAM Capture: User-friendly GUI
• WinPMEM: Kernel-level memory acquisition

2. Virtual Machine Memory:

• VMware: .vmem files
• VirtualBox: .sav files
• Hyper-V: .bin files

Volatility Framework Analysis

Basic Volatility Commands:

# Identify image profile
volatility -f memory.dmp imageinfo

# List processes
volatility -f memory.dmp --profile=Win10x64 pslist

# Network connections
volatility -f memory.dmp --profile=Win10x64 netscan

# Detect injected code
volatility -f memory.dmp --profile=Win10x64 malfind

# Extract executables
volatility -f memory.dmp --profile=Win10x64 procdump -p [PID] -D output/

Identifying Malicious Artifacts

Memory Indicators:

1. Process Anomalies:

• Hidden processes (psxview)
• Suspicious process relationships
• Injected code regions

2. Network Artifacts:

• Active connections
• Listening ports
• DNS cache entries

3. Persistence Mechanisms:

• Registry keys in memory
• Scheduled tasks
• Service configurations

7. Automated Analysis Tools

Sandbox Analysis

Online Sandboxes:

1. Hybrid Analysis:

• Free tier available
• Detailed behavioral reports
• Network traffic analysis
• Screenshot capture

2. Cuckoo Sandbox (Self-hosted):

• Installation and configuration:

# Basic Cuckoo setup
pip install cuckoo
cuckoo init
cuckoo community

• Customizable analysis modules
• Extensive reporting options

3. Joe Sandbox:

• Advanced anti-evasion techniques
• Deep malware analysis
• Detailed PDF reports

YARA Rules

Creating Detection Rules:

rule Suspicious_Process_Injection
{
    meta:
        description = "Detects potential process injection"
        author = "Security Analyst"
        date = "2024-01-01"
    
    strings:
        $api1 = "VirtualAllocEx"
        $api2 = "WriteProcessMemory"
        $api3 = "CreateRemoteThread"
        $api4 = "OpenProcess"
        
    condition:
        uint16(0) == 0x5A4D and
        all of ($api*)
}

YARA Implementation:

1. Rule development based on analysis
2. Testing against sample sets
3. Integration with security tools
4. Continuous rule updates

8. Reporting and Documentation

Analysis Report Structure

Executive Summary:

• Threat classification
• Risk assessment
• Key findings
• Recommended actions

Technical Analysis:

1. Static Analysis Results:

• File properties and metadata
• Code analysis findings
• Identified capabilities

2. Dynamic Analysis Results:

• Behavioral summary
• System modifications
• Network activity

3. Indicators of Compromise (IOCs):

File IOCs:
- SHA256: [hash]
- Filename: malware.exe
- File size: 524288 bytes

Network IOCs:
- C2 Domain: malicious-c2.com
- IP Address: 192.168.1.100
- User-Agent: CustomBot/1.0

Registry IOCs:
- Key: HKLM\Software\Microsoft\Windows\CurrentVersion\Run
- Value: MalwareStartup

Threat Intelligence Sharing

STIX/TAXII Format:

{
  "type": "indicator",
  "pattern": "[file:hashes.SHA256 = 'abc123...']",
  "labels": ["malicious-activity"],
  "valid_from": "2024-01-01T00:00:00.000Z"
}

9. Best Practices and Safety Considerations

Safety Guidelines

1. Always Use Isolated Environments:
   • Never analyze on production systems
   • Maintain network isolation
   • Use dedicated analysis machines
2. Document Everything:
   • Keep detailed analysis notes
   • Screenshot important findings
   • Maintain chain of custody
3. Legal Considerations:
   • Ensure proper authorization
   • Respect privacy laws
   • Follow organizational policies

Continuous Learning

Skill Development:

1. Practice Resources:

• Malware analysis challenges
• CTF competitions
• Online courses and certifications

2. Community Engagement:

• Security forums and blogs
• Threat intelligence sharing
• Conference participation

3. Tool Proficiency:

• Regular tool updates
• Script development
• Automation skills

Anti-Analysis Techniques

Common Evasion Methods:

1. VM Detection:

• Registry key checks
• Hardware fingerprinting
• Timing-based detection

2. Debugger Detection:

• IsDebuggerPresent API
• PEB manipulation
• Exception handling tricks

3. Sandbox Evasion:

• Sleep timers
• User interaction requirements
• Environment fingerprinting

Countermeasures:

• Use bare-metal systems when needed
• Employ anti-analysis tools
• Manual bypassing techniques

Malware analysis is a complex but essential cybersecurity discipline. This guide provides a foundation for understanding and practicing safe malware analysis techniques. Remember that malware analysis should always be performed ethically and legally, with the goal of improving security defenses and protecting systems from threats.

Additional Resources

Tools and Downloads:

• Static Analysis: IDA Free, Ghidra, x64dbg, PE-bear
• Dynamic Analysis: Process Monitor, Process Explorer, API Monitor
• Network Analysis: Wireshark, NetworkMiner, TCPView
• Memory Analysis: Volatility, Rekall, WinDbg
• Sandboxes: Cuckoo, CAPE, Hybrid Analysis

Learning Platforms:

• Practical Malware Analysis book
• Malware Unicorn tutorials
• OpenSecurityTraining courses

Communities:

• /r/Malware subreddit
• Malware Analysis Discord servers
• Twitter security community (#malwareanalysis)
• Local security meetups and conferences

Remember: The goal of malware analysis is to understand threats and build better defenses, not to create or distribute malicious software. Always practice responsible disclosure and ethical analysis.