Page 8 of 8 [ 127 posts ]  Go to page Previous  1 ... 4, 5, 6, 7, 8

Mona Pereth
Veteran
Veteran

Joined: 11 Sep 2018
Gender: Female
Posts: 9,087
Location: New York City (Queens)

07 Jul 2026, 2:24 pm

To jeffj8086: Thanks for all your work.

Some questions:

Who will the moderator(s) be now?

Where's CornFlake?


_________________
- Autistic in NYC - Resources and new ideas for the autistic adult community in the New York City metro area.
- Autistic peer-led groups (via text-based chat, currently) led or facilitated by members of the Autistic Peer Leadership Group.


steve30
Velociraptor
Velociraptor

User avatar

Joined: 16 Feb 2007
Age: 35
Gender: Male
Posts: 482
Location: Rotherham

07 Jul 2026, 6:58 pm

Having a completely new system sounds like a decent idea, but given that the site is over two decades old, I think it would be good to save an archive of the existing site as there is a wealth of information and discussion here - in fact I often search out old topics on certain subjects. Its also a good indicator of how things have changed over time.

I first started using web forums in the early 2000s when they were quite a new thing, and no one really imagined that they would still be lurking around two decades later. Another forum I'm on lost everything pre-2006 as the forum was hosted by Invisionfree who screwed up somewhere and we were unable to bring the database with us to a new host. I also used to be on the 68kMLA forums - they kept their old Snitz forum as a read only archive which is still usable to this day, but a lot of the subsequent phpBB forum (2003-2007) got completely lost in a big server crash in 2007.



Cornflake
Administrator
Administrator

User avatar

Joined: 30 Oct 2010
Gender: Male
Posts: 73,341
Location: Over there

08 Jul 2026, 10:36 am

Mona Pereth wrote:
Where's CornFlake?
*raises hand* Here!


_________________
Giraffe: a ruminant with a view.


Tamaya
Veteran
Veteran

Joined: 8 May 2025
Age: 36
Gender: Female
Posts: 2,668
Location: England

08 Jul 2026, 10:49 am

Cornflake wrote:
Mona Pereth wrote:
Where's CornFlake?
*raises hand* Here!


Cornflake! Welcome back! :cheers:


_________________
My diagnosis story and why it was a traumatic experience for me:
viewtopic.php?f=35&t=416910&start=1056#p9695026

Please notify me if there's a spelling mistake or an obvious autocorrect error in my posts.


exec
Veteran
Veteran

User avatar

Joined: 26 Oct 2024
Gender: Male
Posts: 4,278
Location: USA

08 Jul 2026, 11:30 am

Welcome back :)


_________________
“Success is only meaningful and enjoyable if it feels like your own.” -Michelle Obama


Mona Pereth
Veteran
Veteran

Joined: 11 Sep 2018
Gender: Female
Posts: 9,087
Location: New York City (Queens)

09 Jul 2026, 6:10 pm

Glad to see you're back.


_________________
- Autistic in NYC - Resources and new ideas for the autistic adult community in the New York City metro area.
- Autistic peer-led groups (via text-based chat, currently) led or facilitated by members of the Autistic Peer Leadership Group.


jeffj8086
Butterfly
Butterfly

Joined: 23 Feb 2026
Age: 40
Gender: Male
Posts: 14

09 Jul 2026, 10:16 pm

Hey everyone, I'm sorry. This last issue was my fault. There will be some spotty service for at least a week. Again, I'm sorry.



squeeker
Deinonychus
Deinonychus

User avatar

Joined: 16 Feb 2007
Age: 35
Gender: Female
Posts: 316
Location: Canada

09 Jul 2026, 10:31 pm

No worries Jeff, I think everyone will just be happy to see it again when they log in. :D



exec
Veteran
Veteran

User avatar

Joined: 26 Oct 2024
Gender: Male
Posts: 4,278
Location: USA

10 Jul 2026, 6:40 am

Thank you for letting us know, Jeff :)


_________________
“Success is only meaningful and enjoyable if it feels like your own.” -Michelle Obama


Cornflake
Administrator
Administrator

User avatar

Joined: 30 Oct 2010
Gender: Male
Posts: 73,341
Location: Over there

13 Jul 2026, 9:10 am

Thanks all for the welcome, and thanks Jeff for the under-the-hood debugging work.
It's very good to know that at last we have a responsive point of call for those things.


_________________
Giraffe: a ruminant with a view.


jeffj8086
Butterfly
Butterfly

Joined: 23 Feb 2026
Age: 40
Gender: Male
Posts: 14

14 Jul 2026, 6:55 pm

What the AI demons say about the attack and recovery postmortem:

|# Postmortem: WrongPlanet Exploitation — July 2026

**Date of incident:** July 12–13, 2026
**Date of discovery:** July 13, 2026 09:00 UTC (automated scanner)
**Date of recovery:** July 14, 2026
**Author:** runnel
**Status:** Resolved. Site live, malicious code removed, hardening applied.

---

## Summary

WrongPlanet.net was compromised via a WordPress vulnerability. An attacker
uploaded 8 PHP web shells to `wp-content/uploads/originals/`, established
persistence via malicious mu-plugins and a backdoored plugin file, injected
a Chinese SEO cloaking system, and created a backdoor administrator account.
The wp-security-scanner detected the intrusion on its next daily run. Jeff
took the site down administratively and deployed a holding page. The
filesystem was cleaned, the backdoor account was removed, and significant
nginx/PHP hardening was applied to prevent recurrence.

No database was lost or reverted. All forum posts and user data are intact.

---

## Timeline (UTC)

| Time | Event |
|------|-------|
| **Jul 12 ~09:00** | Last clean scanner run. 0 threats, 0 changes. Site was healthy. |
| **Jul 12 09:00 – Jul 13 09:00** | Attack window. Attacker gained WordPress access (likely via a known WP 3.9.x vulnerability or brute-force of a weak admin password). Uploaded 8 PHP web shells to `wp-content/uploads/originals/` and `wp-content/uploads/`. Modified `wp-content/plugins/wordpress-seo/cache-helper.php` and `wp-content/themes/wrongplanet/functions.php`. |
| **Jul 13 09:00** | Scanner detected: **THREAT** — 8 PHP files in uploads (`avatar50.php`, `valid_test_php.php`, `rce_final.php`, `rce_appended.php`, `idat_payload.php`, `idat_win_L9.php`, `wp_shell.php`, `cache-thumbs-2019.php`). 10 changed files total. ALERT email sent to recipients. |
| **Jul 13 ~10:00** | Initial cleanup began (unclear by whom — possibly Jeff). Web shells removed from uploads. However, new persistence files appeared: `wp-content/mu-plugins/wp-core-update.php`, `wp-content/db.php`, `wp-includes/SimplePie/Parse/Date.php`. |
| **Jul 13 10:22** | Scanner run (likely manual). PHP in uploads: CLEAN. But 3 new changed files detected: `mu-plugins/wp-core-update.php`, `db.php`, `wp-includes/SimplePie/Parse/Date.php`. |
| **Jul 13 ~12:00** | Further modifications: `functions.php`, `wp-includes/load.php`, `wp-config.php` modified. Attacker was actively modifying core files. |
| **Jul 13 12:16** | Scanner run (likely manual). 3 changed files: `functions.php`, `load.php`, `wp-config.php`. |
| **Jul 13 12:16 – Jul 14 09:00** | Attacker installed SEO cloaking mu-plugin (`seo-cache-helper.php`). Earlier persistence files cleaned up by Jeff, but `analytics-verify.php` and `seo-cache-helper.php` remained in mu-plugins. `dynamic-widgets/widget-init.php` was backdoored. |
| **Jul 14 09:00** | Scanner run. 2 changed files: `mu-plugins/seo-cache-helper.php`, `wp-includes/load.php`. No threats (web shells already cleaned). |
| **Jul 14 ~18:00** | Jeff took the site down administratively. Deployed a Caddy reverse proxy + `wrongplanet_holding` nginx container serving a "site down" page on port 53801. Stopped npm, wpnet_web, wpnet_php. |
| **Jul 14 ~23:00** | runnel began recovery. Snapper snapshot taken. Scanner logs reviewed. Attack timeline reconstructed. |

---

## Attack Analysis

### Entry vector

**WordPress 3.9.40 running on PHP 5.6** — both are end-of-life with known
unpatched vulnerabilities. The exact exploit is unknown (no web server
logs from the attack window were preserved in a way that identifies the
initial entry), but likely candidates:

- Known WP 3.9.x REST API or AJAX vulnerabilities allowing arbitrary file upload
- Brute-force of a weak admin password (the `adminlin` account existed since 2012 with email `admin@admin.com`)
- A vulnerable plugin (wordpress-seo, dynamic-widgets, theme-my-login, etc.)

### Persistence mechanisms found

| File | Type | Purpose |
|------|------|---------|
| `wp-content/mu-plugins/analytics-verify.php` | Web shell | `eval(base64_decode($_GET["gclid"]))` gated by cookie `_ga_sess=san666666`. Disguised as "Google Analytics Verification" plugin. |
| `wp-content/mu-plugins/seo-cache-helper.php` | SEO cloaking | Chinese SEO spam injection. Served a Quark browser download page (夸克下载) to search engine crawlers via reverse-DNS-verified Googlebot detection. Redirected human visitors to baidu.com. |
| `wp-content/plugins/dynamic-widgets/widget-init.php` | Web shell | Same `eval(base64_decode($_GET["fbclid"]))` pattern with same `san666666` cookie. Prepended to a fake plugin init file (the real init file is `dynwid_init_worker.php`). |
| `wp-content/mu-plugins/.crawl.log` | Artifact | Log of crawler visits redirected by the SEO cloaking system. |
| `wp-content/mu-plugins/.seo-cache.html` | Artifact | Cached Chinese SEO spam page (Quark browser download). |
| WP user `adminlin` (ID 130659) | Backdoor admin | WordPress administrator account, registered 2012-08-03, email `admin@admin.com`. Unclear if created by original attacker in 2012 or reused later. |
| `wp-includes/load.php` (modified) | Core file modification | Modified by attacker on Jul 13. Restored from clean backup. |
| `wp-config.php` (modified) | Config modification | `WP_DEBUG` set to `true` (possibly to aid debugging during exploitation). Disabled during recovery. |
| 8 PHP files in `wp-content/uploads/originals/` | Web shells | Initial payload: `avatar50.php`, `valid_test_php.php`, `rce_final.php`, `rce_appended.php`, `idat_payload.php`, `idat_win_L9.php`, `wp_shell.php`, `cache-thumbs-2019.php`. Cleaned up before runnel's involvement. |
| `wp-content/mu-plugins/wp-core-update.php` | Persistence | Appeared in Jul 13 10:22 scan. Later removed (unclear by whom). |
| `wp-content/db.php` | Persistence | Appeared in Jul 13 10:22 scan. Later removed. |
| `wp-includes/SimplePie/Parse/Date.php` | Persistence | Appeared in Jul 13 10:22 scan. Later removed. |

### What the attacker achieved

- Arbitrary PHP code execution on the server
- WordPress administrator access (via `adminlin` account)
- SEO cloaking — search engine crawlers were served a Chinese Quark browser
download page instead of the real WrongPlanet content
- Potential data exfiltration (all WP and phpBB databases were accessible
from the PHP environment — no evidence of exfiltration found, but cannot
be ruled out)

### What the attacker did NOT achieve

- No phpBB founder/admin escalation (founders list unchanged in scanner logs)
- No database deletion or corruption (DB was intact when recovery began)
- No lateral movement to other services (Jitsi, scanner, etc. unaffected)
- No shell access to the host (exploit was confined to the PHP/web layer)

---

## Recovery Actions

### Snapshot
- Snapper snapshot #56 (`pre-exploit-recovery-2026-07-14`) taken on root FS
- Snapper snapshot #57 (`pre-hardening-2026-07-14`) taken before hardening

### Filesystem cleanup
1. Removed `wp-content/mu-plugins/analytics-verify.php` (web shell)
2. Removed `wp-content/mu-plugins/seo-cache-helper.php` (SEO cloaking)
3. Removed `wp-content/mu-plugins/.crawl.log` and `.seo-cache.html` (artifacts)
4. Removed `wp-content/plugins/dynamic-widgets/widget-init.php` (backdoored file — not a real plugin file)
5. Restored `wp-includes/load.php` from clean backup at `/home/toggi3/wpnet/site/`
6. Disabled `WP_DEBUG` in `wp-config.php` (was enabled by attacker)
7. Confirmed `wp-content/mu-plugins/0-worker.php` is legitimate ManageWP loader (kept)
8. Flushed nginx fastcgi cache (was serving cached Chinese SEO page)

### Database cleanup
1. Deleted WP user `adminlin` (ID 130659) — removed from `wp_users` and `wp_usermeta`
2. Verified no `adminlin` in phpBB `nuke_bbusers`
3. No other DB changes made — all forum posts, users, and data preserved

### Service restoration
1. Stopped `wrongplanet_holding` container (holding page)
2. Stopped `caddy` container (temporary reverse proxy)
3. Started `npm` (nginx-proxy-manager)
4. Started `wpnet_web`, `wpnet_php`, `wpnet_phpmyadmin` (WP stack)
5. Verified: homepage 200, forums 200, topic pages 200

### Verification
- Grep for backdoor signatures (`san666666`, `_ga_sess.*eval`, `gclid.*eval`): CLEAN
- PHP files in uploads: CLEAN (only `index.php` in `profiles/` which is a 404 stub)
- Scanner rerun: detected our cleanup changes, no remaining threats

---

## Hardening Applied

### 1. File integrity diffing (scanner UNIT 15)
New scanner unit compares current "Recent file changes" against previous
run and reports MD5 hashes of changed files. Shows *what* changed, not just
*that* something changed.

### 2. Web shell access log monitoring (scanner UNIT 16)
New scanner unit greps npm access logs for known web shell patterns:
`_ga_sess.*san666`, `gclid`/`fbclid` with long base64 payloads, known shell
filenames (`wp-shell`, `rce_final`, `c99`, `b374k`, `filesman`, `phpspy`),
SEO spam filenames (`cache-thumbs-20`). Triggers THREAT on match.
Scanner docker-compose now mounts npm logs at `/npm_logs:ro`.

### 3. PHP error log monitoring (scanner UNIT 17)
New scanner unit reads `wp-content/debug.log` incrementally (using stored
offset from previous run) and reports new PHP warnings, fatal errors, and
MySQL connection failures. Triggers WARNING severity.
Scanner docker-compose now mounts debug log at `/wp_debug.log:ro`.

### 4. Snapper coverage for /opt
New snapper config created for `/opt` subvolume (where `compose_stacks`
lives). First snapshot: `initial opt snapshot - post exploit cleanup`.
Future changes to webroot, container configs, and scanner files are now
covered by snapper.

### 5. nginx hardening — WordPress-specific
Complete restructure of `nginx.conf` with independent location blocks:

**phpBB (`/forums/`)** — unrestricted, serves itself:
- PHP execution allowed for all `.php` files
- Same caching rules as before
- No WordPress restrictions applied

**WordPress (everything else)** — locked down:
- Only `index.php` and `wp-login.php` can execute PHP
- `wp-config.php` → 403
- `xmlrpc.php` → 403
- `wp-admin/` → 403
- `wp-json/` (REST API) → 403
- `wp-includes/*.php` → 403
- `wp-content/uploads/*.php` → 403
- `wp-content/*.php` (direct files) → 403
- Hidden files (`.htaccess`, `.git`, etc.) → 403
- Catch-all `~ \.php$` → 403 (any PHP file not explicitly allowed)

### 6. PHP hardening
Added `ini_restore` and `putenv` to `disable_functions` in `php.ini`.
These are commonly used by attackers to override security settings at
runtime. Full list now: `shell_exec, proc_open, popen, symlink, link,
posix_getpwuid, posix_getgrgid, posix_getpid, posix_uname, ini_restore,
putenv`.

Kept `exec`, `system`, `passthru` enabled — phpBB needs them (nslookup,
ImageMagick).

### Backups saved
- `nginx.conf.bak.pre-hardening`
- `php.ini.bak.pre-hardening`

---

## Gaps and Recommendations

### What worked
- **Scanner detected the attack** within 24 hours. The daily scan at 09:00
caught the 8 PHP web shells in uploads and triggered an ALERT email.
- **Database was preserved** — no data loss.
- **Scanner logs provided the full attack timeline** — every change was
documented with timestamps.

### What didn't work
- **Nobody acted on the ALERT email for ~18 hours.** The scanner sent the
alert at 09:00 but the site wasn't taken down until ~18:00. The attacker
had a full day of additional access after detection.
**Recommendation:** Configure the scanner to send an SMS or push
notification on ALERT severity, not just email. Or configure a check
that pages if an ALERT-severity scan result hasn't been acknowledged.

- **Snapper didn't cover /opt.** The webroot lives under
`/opt/compose_stacks/wrongplanet_production/site/` which is on a separate
btrfs subvolume. Snapper only snapshotted the root FS. We couldn't
restore from snapshots — had to use a static backup from 2014.
**Status: Fixed.** Snapper now covers /opt.

- **nginx fastcgi cache served the Chinese SEO page after cleanup.** The
malicious page was cached and continued to be served even after the
PHP code was removed. Required a manual cache flush.
**Recommendation:** The scanner or a post-cleanup script should flush
the nginx cache. Or configure `fastcgi_cache_valid` with a shorter TTL.

- **No initial access log retention for forensics.** The attacker's initial
entry vector is unknown because we don't have access logs from the attack
window in a form that identifies the exploit request. npm logs are
rotated and compressed.
**Recommendation:** Increase npm access log retention to 30 days minimum.
Consider forwarding access logs to a remote syslog server.

- **WordPress 3.9.40 on PHP 5.6 is fundamentally insecure.** Both are EOL
with unpatched vulnerabilities. This attack will recur unless WordPress
is removed or upgraded.
**Recommendation:** Execute the decoupling plan
(`~/plans/2026-07-06-decouple-phpbb-from-wordpress.md`) to remove
WordPress entirely. The nginx hardening significantly reduces the attack
surface but doesn't eliminate it — `wp-login.php` and `index.php` are
still exposed, and a vulnerable WP core can be exploited through them.

- **The `adminlin` backdoor admin existed since 2012.** It's unclear when
it was created or if it was the entry vector for this attack or a prior
one.
**Recommendation:** Audit all WP admin accounts periodically. The
scanner already checks phpBB founders/admins but doesn't check WP admin



squeeker
Deinonychus
Deinonychus

User avatar

Joined: 16 Feb 2007
Age: 35
Gender: Female
Posts: 316
Location: Canada

14 Jul 2026, 8:42 pm

Oh wow, thanks for the update and for working on getting the site and forums more secure. It was interesting to read through even though I don't really understand.



exec
Veteran
Veteran

User avatar

Joined: 26 Oct 2024
Gender: Male
Posts: 4,278
Location: USA

15 Jul 2026, 6:41 am

Wowzers ... just so happy we recovered and are back up now. Much thanks to the staff here for the hard work. :)


_________________
“Success is only meaningful and enjoyable if it feels like your own.” -Michelle Obama


MjrMajorMajor
Veteran
Veteran

User avatar

Joined: 15 Jan 2012
Gender: Female
Posts: 8,816

Yesterday, 7:51 pm

exec wrote:
Wowzers ... just so happy we recovered and are back up now. Much thanks to the staff here for the hard work. :)


100%!



CockneyRebel
Veteran
Veteran

User avatar

Joined: 17 Jul 2004
Age: 51
Gender: Male
Posts: 121,235
Location: In my own little country

Today, 12:45 pm

I'm glad we sprung back up


_________________
The Family Schlager