diff options
| author |  Inverle <inverle@proton.me>
| 2025-06-22 00:09:18 +0200 |
|---|---|---|
| committer |  GitHub <noreply@github.com>
| 2025-06-22 00:09:18 +0200 |
| commit | 18b5c8ec6da64da90706643ffa52736a85a2ca59 (patch) | |
| tree | 0b7ac0082c0a0aa86c0f7a4caccb616011576023 /app/Models/Entry.php | |
| parent | a6948218fb1c66fe146c7651555e5a1f791c8112 (diff) | |
Handle redirects when scraping feed from HTML (#7654)
* Handle redirects when scraping feed from HTML
* pass codesniffer
* pass PHPStan
* Optimize
* Another approach relying on HTML base
Standard way to save an HTML document with relative references
* Fix case of existing HTML base
which should not be overriden
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
Diffstat (limited to 'app/Models/Entry.php')
| -rw-r--r-- | app/Models/Entry.php | 10 |
1 files changed, 7 insertions, 3 deletions
diff --git a/app/Models/Entry.php b/app/Models/Entry.php index bc5ed2279..66c05a830 100644 --- a/app/Models/Entry.php +++ b/app/Models/Entry.php @@ -845,7 +845,7 @@ HTML; * @param string $url Overridden URL. Will default to the entry URL. * @throws Minz_Exception */ - public function getContentByParsing(string $url = '', int $maxRedirs = 3): string { + public function getContentByParsing(string $url = '', int $maxRedirs = 4): string { $url = $url ?: htmlspecialchars_decode($this->link(), ENT_QUOTES); $feed = $this->feed(); if ($url === '' || $feed === null || $feed->pathEntries() === '') { @@ -872,12 +872,16 @@ HTML; } $cachePath = $feed->cacheFilename($url . '#' . $feed->pathEntries()); - $html = httpGet($url, $cachePath, 'html', $feed->attributes(), $feed->curlOptions()); - if (strlen($html) > 0) { + $response = httpGet($url, $cachePath, 'html', $feed->attributes(), $feed->curlOptions()); + $html = $response['body']; + if ($html !== '') { $doc = new DOMDocument(); $doc->loadHTML($html, LIBXML_NONET | LIBXML_NOERROR | LIBXML_NOWARNING); $xpath = new DOMXPath($doc); + // Account for HTTP redirections + $url = $response['effective_url'] ?: $url; + $maxRedirs -= $response['redirect_count']; if ($maxRedirs > 0) { //Follow any HTML redirection $metas = $xpath->query('//meta[@content]') ?: []; |